Question

I am after smooth texture based outline effect in OpenGL. So far I tried mostly all kinds of edge detection algorithms which result mostly in crude and jagged outlines. Then I read about Distance Field. I found an example which does pretty nice distance field. Here is the GLSL code:

#version 420
layout(binding=0)  uniform sampler2D colorMap;
flat in vec4 diffuseOut;
in vec2 uvsOut;
out vec4 outputColor;
const float ALPHA_THRESHOLD = 0.9;
const float NUM_SPOKES = 36.0; // Number of radiating lines to check in.
const float ANGULAR_STEP =360.0 / NUM_SPOKES;
const int ZERO_VALUE =128; // Color channel containing 0 => -128, 128 => 0, 255 => +127
int in_StepSize=15;    // Distance to check each time (larger steps will be faster, but less accurate).
int in_MaxDistance=30; // Maximum distance to search out to. Cannot be more than 127!

vec4 distField(){

    vec2 pixel_size = 1.0 / vec2(textureSize(colorMap, 0));
    vec2 screenTexCoords = gl_FragCoord.xy * pixel_size;
    int distance;

    if(texture(colorMap, screenTexCoords).a == 0.0)
    {
        // Texel is transparent, search for nearest opaque.
        distance = ZERO_VALUE + 1;
        for(int i = in_StepSize; i < in_MaxDistance; i += in_StepSize)
        {
            if(find_alpha_at_distance(screenTexCoords, float(i) * pixel_size, 1.0))
            {
                i = in_MaxDistance + 1; // BREAK!
            }
            else
            {
                distance = ZERO_VALUE + 1 + i;
            }
        }
    }
    else
    {
        // Texel is opaque, search for nearest transparent.
        distance = ZERO_VALUE;
        for(int i = in_StepSize; i <= in_MaxDistance; i += in_StepSize)
        {
            if(find_alpha_at_distance(screenTexCoords, float(i) * pixel_size, 0.0))
            {
                i = in_MaxDistance + 1; // BREAK!
            }
            else
            {
                distance = ZERO_VALUE - i;
            }
        }
    }

    return  vec4(vec3(float(distance) / 255.0) * diffuseOut.rgb, 1.0 - texture(colorMap, screenTexCoords).a);

}

void main()
{
    outputColor= distField();
}

The result of this shader covers the whole screen using the diffuse color for filling the screen area outside the Distance Field outline.Here is how it looks like :

What I need is to leave all the area which has the solid red fill outside the distance field as transparent.

Answer 1

I came to the solution by using Distance Field gray scale 8 bit alpha map.Stefan Gustavson
describes in detail how to do it.Basically one needs to generate the distance field version of the original texture.Then this texture is rendered with the primitive normally in the first pass into an FBO.In the second pass the alpha blending mode should be on.The texture from the first pass in used with the screen quad.At this stage the the fragment shader samples the alpha from that texture.This results in both smooth edges and alpha transparency around the edges.
Here is the result: