Question

I have completely re-written this first post to better show what the problem is.

I am using ps v1.4 (highest version I have support for) and keep getting an error.
It happens any time I use any type of function such as cos, dot, distance, sqrt, normalize etc. on something that was passed into the pixelshader.
For example, I need to do “normalize(LightPosition – PixelPosition)” to use a point light in my pixelshader, but normalize gives me an error.
Some things to note-
I can use things like pow, abs, and radians with no error.
There is only an error if it is done on something passed from the vertex shader. (For example I could take the sqrt of a local pixelshader variable with no error)
I get the error from doing a function on ANY variable passed in, even text coords, color, etc.
Inside the vertex shader I can do all of these functions on any variables passed in with no errors, it’s only in the pixelshader that I get an error
All the values passing from the vertex to pixel shader are correct, because if I use software processing rather than hardware I get no error and a perfectly lit scene.
Since normalizing the vector is essentially where my error comes form I tried creating my own normalizing function.
I call Norm(LightPosition – PixelPosition) and “Norm” looks like this –

float3 Norm(float3 v)
{
    return v / sqrt(dot(v, v));
}

I still get the error because I guess technically I’m still trying to take a sqrt inside the pixelshader.

The error isn’t anything specific, it just says “error in application” on the line where I load my .fx file in C#
I’m thinking it could actually be a compiling error because I have to use such old versions (vs 1.1 and ps 1.4)
When debugged using fxc.exe it tells me “can not map instruction to pixel shader instruction set”

Answer 1

Old GPU:s didn’t always support any instruction, especially in the pixel shader.

You might get away with a sqrt in the vertex shader but for a so old version (1.1 !!) the fragment shader might be extremely limited.

I.e this might not be a bug.

The work around could be to skip the hlsl and write your own assembler (but you might stumble onto the same problem there) and simulate the sqrt (say with a texture lookup and / or interpolations if you can have 2 textures in 1.0 :-p )

You can of course try to write a sqrt-lookup/interpolation in hlsl but it might be too big too (I don’t remember but IIRC 1.1 don’t let you write very long shaders).