I have a 2D node scene graph that I'm trying to 'nest' stencil clipping in.
I was thinking what I could do is when drawing the stencil, increment any pixel it writes to by 1, and keep track of what the current 'layer' is that I'm on.
Then when drawing, only write pixel data to the color buffer if the value of the stencil at that pixel is >= the current layer #.
This is the code I have now. It doesn't quite work. Where am I messing up?
First I call SetupStencilForMask(). Then draw stencil primitives. Next, call SetupStencilForDraw(). Now draw actual imagery When done with a layer, call DisableStencil().
Edit: Updated with solution. It doesn't work for individual items on the same layer, but otherwise is fine. Found a great article on how to actually pull this off, although it's fairly limited. http://cranialburnout.blogspot.com/2014/03/nesting-and-overlapping-translucent.html
// glClear(GL_STENICL_BIT) at start of each draw frame
static int stencilLayer = 0;
void SetupStencilForMask(void)
{
if (stencilLayer == 0)
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_LESS, stencilLayer, 0xff);
glStencilOp(GL_INCR, GL_KEEP, GL_KEEP);
glStencilMask(0xff);
if (stencilLayer == 0)
glClear(GL_STENCIL_BUFFER_BIT);
stencilLayer++;
}
void SetupStencilForDraw()
{
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, stencilLayer, 0xff);
glStencilMask(0x00);
}
void DisableStencil(void)
{
if (--stencilLayer == 0)
glDisable(GL_STENCIL_TEST);
}