Most shadow maps produce waving edges but the trick is to make them so small that they are not noticed during normal game play. Mine were unfortunately very noticable in some situations!
I have plenty of other things to do and should have left it but I could not and had to make an attempt at fixing it.
My first attempt was to change to using variance shadow maps (VSM) instead of single depth based with percentage closer filtering (PCF).
VSM uses both the depth and the depth squared and some maths to minimise artifacts caused by the depth buffer being of insufficient resolution. It avoids the need for a shadow depth bias value. Much to my surprise it was easy to drop in to my most recent shaders. VSM works very well. I am unlikely to go back to PCF.
VSM did not solve my problem. It removed other artifacts and gave me a much crisper edge but that margin was still stretched and unsightly!
As a quick fix, I used the edge tap smoothing code from my PCF methods. That worked to reduce the sharp edges but the margins were still too big. The solution, which I could have also done with my PCF code was to increase the number of samples. I was using 3x3 samples, I tried 5x5 samples.
Unfortunately there was still a problem... When I tried it on the Xbox, it could not cope!
The frame rate dropped a bit but worse it just stopped texturing some surfaces! Aaaaah!
I dropped it to 4x4 sampling, that worked on the Xbox but I was not happy with the shadow margin again, too pixelated! I had and still have loads of time consuming things I could try to impliment but I had another problem to solve which has given me acceptable results.
The concave objects that were causing me problems had light bleeding near all the inner edges. I wanted to pull the shadow towards the corner to use more of the dark area and less of the light area. To do this I wanted to offset the area used for the weighting so it was not central but based on the colour away from the light.
Much to my surprise this solved two problems. My 5x5 samples became 5x3, 25 down to only 15 samples, just about workable on the Xbox. Plus the light in the corners of concave objects was reduced to a more acceptable level.
I am not claiming that the results are perfect. I am learning this as I go along, so there is little hope that I will ever get to AAA game levels of detail. The results are now acceptable to me although the frame rate on the Xbox is still a little poor. The wavy edges are no longer distracting during normal game play.
The end result is still a bit stretched and the offset sampling results in an uneven blurred edge but this is for a fast moving shooter not an atmospheric spooky adventure.
I have to have two different quality shaders to get an acceptable frame rate. A lower number of samples on the terrain (3x3) and a higher number of samples on the structures (5x3) to get the better stretched vertical margin.
You have to get pretty close to see all this and if you did that in a game someone would have shot you by now :-)
Now I just need to try blurring the VSM instead...
http://developer.download.nvidia.com/SDK/10/direct3d/Source/VarianceShadowMapping/Doc/VarianceShadowMapping.pdf
Posts
