Question

This is an HLSL question, although I’m using XNA if you want to reference that framework in your answer.

In XNA 4.0 we no longer have access to DX9’s AlphaTest functionality.

I want to:

1. Render a texture to the backbuffer, only drawing the opaque pixels of the texture.
2. Render a texture, whose texels are only drawn in places where no opaque pixels from step 1 were drawn.

How can I accomplish this? If I need to use clip() in HLSL, how to I check the stencilbuffer that was drawn to in step 1, from within my HLSL code?

So far I have done the following:

_sparkStencil = new DepthStencilState
{
    StencilEnable = true,
    StencilFunction = CompareFunction.GreaterEqual,
    ReferenceStencil = 254,
    DepthBufferEnable = true
};


DepthStencilState old = gd.DepthStencilState;
gd.DepthStencilState = _sparkStencil;

// Only opaque texels should be drawn.
DrawTexture1();

gd.DepthStencilState = old;

// Texels that were rendered from texture1 should
// prevent texels in texture 2 from appearing.
DrawTexture2();

Answer 1

To write to the stencil buffer you must create a DepthStencilState that writes to the buffer, then draw any geometry that is to be drawn to the stencil buffer, then switch to a different DepthStencilState that uses the relevant CompareFunction.

If there is some limit on which alpha values are to be drawn to the stencil buffer, then use a shader in the first pass that calls the clip() intrinsic on floor(alpha - val) - 1 where val is a number in (0,1) that limits the alpha values drawn.

I have written a more detailed answer here:

Stencil testing in XNA 4