Question

Description:

On a C# ASP.Net web application, we have implemented some timers to periodically run background tasks. One of the timers occasionally seems to get “doubled” or more rarely “tripled”.

The timer is set to run once every minute and seems to run properly for a while. Eventually, however, it seems like a second timer gets started and calls the timed process a second time within the same time interval. I’ve even seen a case where we had three processes running.

Since this process locks some database records and having a second (or third) process doing the same thing will cause a deadlock or timeout error on the database connection, we’ve implemented a mechanism to only allow one thread at a time to execute the database critical portion of the process code. When the process takes longer than a minute to run, this mechanism successfully blocks the next run triggered by its own timer. But the thread locking fails if the process is triggered by the second (or third) timer.

In our logs, I output both the Process ID and the Managed Thread ID, which lets me see which thread is starting, finishing, or erring out. The strange thing, is that regardless of which timer instance kicked off the process, the Process ID is the same.

var processID = System.Diagnostics.Process.GetCurrentProcess().Id;
var thread = System.Threading.Thread.CurrentThread.ManagedThreadId;

How do I prevent multiple instances of the timer?

We have a web-farm with 2 servers behind a load balancer. I’ve been assurred that the web-garden is set to only allow one instance of the app-pool on each server. A web.config setting specifies which server will run the timed process. The other server will not load the timer.

Relevant Code:

On the Global.asax.cs

protected static WebTaskScheduler PersonGroupUpdateScheduler
{
    get;
    private set;
}

protected void StartSchedulers()
{
    using (var logger = new LogManager())
    {
        // ... other timers configured in similar fashion ...

        if (AppSetting.ContinuousPersonGroupUpdates)
        {
            // clear out-of-date person-group-updater lock
            logger.AppData.Remove("PersonGroupUpdater"); // database record to prevent interference with another process outside the web application.

            var currentServer = System.Windows.Forms.SystemInformation.ComputerName;
            if (currentServer.EqualsIngoreCase(AppSetting.ContinuousPersonGroupUpdateServer))
            {
                PersonGroupUpdateScheduler = new WebTaskScheduler() {
                    AutoReset = true,
                    Enabled = true,
                    Interval = AppSetting.ContinuousPersonGroupUpdateInterval.TotalMilliseconds,
                    SynchronizingObject = null,
                };
                PersonGroupUpdateScheduler.Elapsed += new ElapsedEventHandler(DistributePersonGroupProcessing);
                PersonGroupUpdateScheduler.Start();
                HostingEnvironment.RegisterObject(PersonGroupUpdateScheduler);
                logger.Save(Log.Types.Info, "Starting Continuous Person-Group Updating Timer.", "Web");
            }
            else
            {
                logger.Save(Log.Types.Info, string.Format("Person-Group Updating set to run on server {0}.", AppSetting.ContinuousPersonGroupUpdateServer), "Web");
            }
        }
        else
        {
            logger.Save(Log.Types.Info, "Person-Group Updating is turned off.", "Web");
        }
    }
}

private void DistributePersonGroupProcessing(object state, ElapsedEventArgs eventArgs)
{
    // to start with a clean connection, create a new data context (part of default constructor)
    // with each call.
    using (var groupUpdater = new GroupManager())
    {
        groupUpdater.HttpContext = HttpContext.Current;
        groupUpdater.ContinuousGroupUpdate(state, eventArgs);
    }
}

On a separate file, we have the WebTaskScheduler class which just wraps System.Timers.Timer and implements the IRegisteredObject interface so that IIS will recognize the triggered process as something it needs to deal with when shutting down.

public class WebTaskScheduler : Timer, IRegisteredObject
{
    private Action _action = null;
    public Action Action
    {
        get
        {
            return _action;
        }
        set
        {
            _action = value;
        }
    }

    private readonly WebTaskHost _webTaskHost = new WebTaskHost();

    public WebTaskScheduler()
    {
    }

    public void Stop(bool immediate)
    {
        this.Stop();
        _action = null;
    }
}

Finally, the locking mechanism for the critical section of the code.

public void ContinuousGroupUpdate(object state, System.Timers.ElapsedEventArgs eventArgs)
{
    var pgUpdateLock = PersonGroupUpdaterLock.Instance;

    try
    {
        if (0 == Interlocked.Exchange(ref pgUpdateLock.LockCounter, 1))
        {
            if (LogManager.AppData["GroupImporter"] == "Running")
            {
                Interlocked.Exchange(ref pgUpdateLock.LockCounter, 0);
                LogManager.Save(Log.Types.Info, string.Format("Group Import is running, exiting Person-Group Updater.  Person-Group Update Signaled at {0:HH:mm:ss.fff}.", eventArgs.SignalTime), "Person-Group Updater");
                return;
            }

            try
            {
                LogManager.Save(Log.Types.Info, string.Format("Continuous Person-Group Update is Starting.  Person-Group Update Signaled at {0:HH:mm:ss.fff}.", eventArgs.SignalTime), "Person-Group Updater");
                LogManager.AppData["PersonGroupUpdater"] = "Running";

                // ... prep work is done here ...

                try
                {
                    // ... real work is done here ...

                    LogManager.Save(Log.Types.Info, "Continuous Person-Group Update is Complete", "Person-Group Updater");

                }
                catch (Exception ex)
                {
                    ex.Data["Continuous Person-Group Update Activity"] = "Processing Groups";
                    ex.Data["Current Record when failure occurred"] = currentGroup ?? string.Empty;
                    LogManager.Save(Log.Types.Error, ex, "Person-Group Updater");
                }

            }
            catch (Exception ex)
            {
                LogManager.Save(Log.Types.Error, ex, "Person-Group Updater");
            }
            finally
            {
                Interlocked.Exchange(ref pgUpdateLock.LockCounter, 0);
                LogManager.AppData.Remove("PersonGroupUpdater");
            }

        }
        else
        {
            // exit if another thread is already running this method
            LogManager.Save(Log.Types.Info, string.Format("Continuous Person-Group Update is already running, exiting Person-Group Updater.  Person-Group Update Signaled at {0:HH:mm:ss.fff}.", eventArgs.SignalTime), "Person-Group Updater");
        }
    }
    catch (Exception ex)
    {
        Interlocked.Exchange(ref pgUpdateLock.LockCounter, 0);
        LogManager.Save(Log.Types.Error, ex, "Person-Group Updater");
    }
}

Answer 1

IIS can/will host multiple AppDomains under a worker process (w3wp). These AppDomains can’t/don’t/shouldn’t really talk to each. It’s IIS’s responsibility to manage them.

I suspect what’s happening is that you have multiple AppDomains loaded.

That said…just to be 100% sure…the timer is being started under Application_Start in your global.asax, correct? This will get executed once per AppDomain (not per HttpApplication, as it’s name suggests).

You can check how many app domains are running for your process by using the ApplicationManager's GetRunningApplications() and get GetAppDomain(string id) methods.

In theory you could also do some inter-appdomain communication in there to make sure your process only starts once…but I’d strongly advise against it. In general, relying on scheduling from a web application is ill advised (because your code is meant to be ignorant of how IIS manages your application lifetime).

The preferred/recommended approach for scheduling is via a Windows Service.