Question

We’ve got a long-running process that is initiated by a web request. In order to give the process time to complete, we spin it off on a new thread and use a Mutex to ensure only one instance of the process can run. This code runs as intended in our development and staging environments, but is failing in our production environment with Null Reference Exception. Our application logging does not capture anything and our operations folks are reporting that it is crashing the AppPool. (It would seem to be an environmental problem, but we have to proceed with the assumption that the environments are configured identically.) We have so far been unable to determine where the Null Reference is.

Here is the exception from the Application Event Log:

Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace:    at Jobs.LongRunningJob.DoWork()
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

And here is the code (slightly sanitized):

public class LongRunningJob: Job
{
    private static Mutex mutex = new Mutex();

    protected override void PerformRunJob()
    {
        var ts = new ThreadStart(LongRunningJob.DoWork);
        var thd = new Thread(ts);
        thd.IsBackground = true;
        thd.Start();
    }

    private static void DoWork()
    {
        var commandTimeOut = 180;

        var from = DateTime.Now.AddHours(-24);
        var to = DateTime.Now;

        if (mutex.WaitOne(TimeSpan.Zero))
        {
            try
            {
               DoSomethingExternal(); // from what we can tell, this is never called
            }
            catch (SqlException sqlEx)
            {
                if (sqlEx.InnerException.Message.Contains("timeout period elapsed"))
                {
                    Logging.LogException(String.Format("Command timeout in LongRunningJob: CommandTimeout: {0}", commandTimeOut), sqlEx);
                }
                else
                {
                    Logging.LogException(String.Format("SQL exception in LongRunningJob: {0}", sqlEx.InnerException.Message), sqlEx);
                }
            }
            catch (Exception ex)
            {
                Logging.LogException(String.Format("Error processing data in LongRunningJob: {0}", ex.InnerException.Message), ex);
            }
            finally
            {
                mutex.ReleaseMutex();
            }
        }
        else
        {
            Logging.LogMessage("LongRunningJob is already running.");
        }
    }
}

Answer 1

In order to find a NullReferenceException you basically examine every dereference operation. I can see only the following suspicious one:

ex.InnerException.Message

You can’t assume ex.InnerException is not null.