Question

I am using fxc.exe utility from directx sdk to compile my shaders and then load them as byte array in my application (with this code):

inline std::vector<char> ReadToByteArray(char* filename)
    {
        std::ifstream file;
        file.open(filename);
        file.seekg(0, std::ios::end);
        size_t file_size = file.tellg();
        file.seekg(0, std::ios::beg);
        std::vector<char> data(file_size, 0);
        file.read(&data[0], file_size);

        return data;
    }

And then create it:

std::vector<char> pixelShaderBytecode = ReadToByteArray("myps.pso");
ID3D11PixelShader* shader;
device->CreatePixelShader(pixelShaderBytecode.data(), pixelShaderBytecode.size(), nullptr, &shader);

And all worked good. Until my shader had grown to ~980 bytes in size. Here is my code in HLSL:

struct PInput
{
    float4 position: SV_POSITION;
    float3 normal : NORMAL;
};

cbuffer CBuffer
{
    float4 color;
    float4 dirs[8];
};

float4 main(PInput input) : SV_TARGET
{
    // Example code...

    int directionLightsCount = 2;

    float intensity = 0.0f;

    // Line 1
    intensity += saturate(dot(input.normal, -dirs[0].xyz));

    // Line 2
    //intensity += saturate(dot(input.normal, -dirs[1].xyz));

    float ambient = 0.1f;

    return color * saturate(ambient + intensity);
}

Problem is, that if i will uncomment line 2, then i will get E_INVALIDARG HRESULT from ID3D11Device::CreatePixelShader method. And also warning from D3D11 debug layer:

D3D11 ERROR: ID3D11Device::CreatePixelShader: Pixel Shader is corrupt or in an unrecognized format. [ STATE_CREATION ERROR #192: CREATEPIXELSHADER_INVALIDSHADERBYTECODE]

If i will comment this line again, then all will work. If i will change them (0 index with 1) then all will work. The only difference i see – size of compiled file (980 with both lines, and 916 with just first line)

Here is the command with which i compile hlsl code:

C:\Program Files (x86)\Microsoft DirectX SDK (June 2010)\Utilities\bin\x64>fxc /
T ps_5_0 /E "main" /Fo "ps.pso" ps.ps

What is the reason of this behavior? Am i missed something important?

Thanx for any help.

EDIT: Forgot to say, that both versions compiled without any errors. Also, vertex shader created without any problem (1400 bytes). I am using 11_0 feature level.

Answer 1

A simple error: wrong file reading…

inline std::vector<char> ReadToByteArray(char* filename)
{
    std::vector<char> data;
    std::ifstream file(filename, std::ios::in | std::ios::binary | std::ios::ate);
    data.resize(file.tellg());
    file.seekg(0, std::ios::beg);
    file.read(&data[0], data.size());
    return data;
}