Question

I need help with a simple spotlight shader.
All vertices inside the cone should be colored yellow, all vertices outside the cone should be colored black.
I just can’t get it work. I asume it has something to do with the transformation from world into eye coordinates.

Vertex Shader:

uniform vec4  lightPositionOC;   // in object coordinates
uniform vec3  spotDirectionOC;   // in object coordinates
uniform float spotCutoff;        // in degrees

void main(void)
{
   vec3 lightPosition;
   vec3 spotDirection;
   vec3 lightDirection;
   float angle;

    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;

    // Transforms light position and direction into eye coordinates
    lightPosition  = (lightPositionOC * gl_ModelViewMatrix).xyz;
    spotDirection  = normalize(spotDirectionOC * gl_NormalMatrix);

    // Calculates the light vector (vector from light position to vertex)
    vec4 vertex = gl_ModelViewMatrix * gl_Vertex;
    lightDirection = normalize(vertex.xyz - lightPosition.xyz);

    // Calculates the angle between the spot light direction vector and the light vector
    angle = dot( normalize(spotDirection),
                -normalize(lightDirection));
    angle = max(angle,0);   

   // Test whether vertex is located in the cone
   if(angle > radians(spotCutoff))
       gl_FrontColor = vec4(1,1,0,1); // lit (yellow)
   else
       gl_FrontColor = vec4(0,0,0,1); // unlit(black)   
}

Fragment Shader:

void main(void)
{
   gl_FragColor = gl_Color;
}

Edit:
Tim is right. This

if(angle > radians(spotCutoff))

should be:

if(acos(angle) < radians(spotCutoff))

New question:
The light seems not to stay at a fixed position in the scene, instead it seems to move relative to my camera as the cone gets smaller or bigger when I move forward or backward.

Answer 1

(Let spotDirection be vector A, and lightDirection be vector B)

You are assigning;

angle = dot(A,B)

Shouldn’t the formula be:

cos(angle) = dot(A,B)

or

angle = arccos(dot(A,B))

http://en.wikipedia.org/wiki/Dot_product#Geometric_interpretation