I bumped into a problem with normal mapping and surfaces pointing away from light. These surfaces should not be lit at all since they are not facing the light. However, it is possible that quite a few fragments from these surfaces will be lit if fragments’ normals happen to be bent enough. Artifacts caused by this are clearly visible e.g. on a box which has one side pointing directly away from light: almost all fragments of that side are black (as they are supposed to) but still couple of white fragments are visible (sorry, screenshot is not currently available but if you have done normal mapping before you should know what I am talking about).
I found a simple way to fix this in forward renderer: just multiply light intensity by (4*lightDirection.Z). Reasoning is that you should multiply light intensity by (clamped) dot product of light vector and surface’s normal vector (if face is pointing away, dot product will be negative and light intensity will be clamped to 0), and since we are doing calculations in tangent space, this normal is always (0,0,1) and thus dot product is simply lightDirection.z. Multiplier 4 is there just to make sure that light intensity will not drop (if face is not pointing away).
Now I tried to implement same functionality to deferred renderer but noticed soon that this simple approach will not work since I am doing calculations in world space. To do this calculation I would need both real normal of that fragment AND normal of the surface on which this fragment is located. Of course I could store this information to a texture but spending 3 more floats (or 2 if I was to pack them) just to do this trick seems quite expensive.
Are there any workarounds?
Thank you.
In deferred shading you store normal in view space, so I think you just need to transform your light direction from world space to view space first, then you can do the same calculation.