I have a worker role that works perfectly in development but doesn’t work when deployed. “Doesn’t work” is rather vague, but that’s really all I have to go on as I’m not seeing any errors or anything (in the event log anyway – maybe there is somewhere else I can look). I have added some trace statements to my code and I’m seeing the first one come out, but none of the others.
WorkerRole Code:
public class WorkerRole : RoleEntryPoint
{
#region Member variables
private IWindsorContainer _container;
private IJob[] _jobs;
#endregion
#region Methods
public override bool OnStart()
{
ConfigureDiagnostics();
Trace.WriteLine("WorkerRole.OnStart()");
try
{
Initialize();
Trace.WriteLine("Resolving jobs...");
_jobs = _container.ResolveAll<IJob>();
StartJobs();
return base.OnStart();
}
catch (Exception ex)
{
TraceUtil.TraceException(ex);
throw;
}
finally
{
Trace.WriteLine("WorkerRole.OnStart - Complete");
Trace.Flush();
}
}
/// <summary>
/// Sets up diagnostics.
/// </summary>
private void ConfigureDiagnostics()
{
DiagnosticMonitorConfiguration dmc =
DiagnosticMonitor.GetDefaultInitialConfiguration();
dmc.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
DiagnosticMonitor.Start(Constants.DiagnosticsConnectionString, dmc);
}
/// <summary>
/// Sets up the IoC container etc.
/// </summary>
private void Initialize()
{
Trace.WriteLine("WorkerRole.Initialize()");
try
{
Trace.WriteLine("Configuring AutoMapper...");
AutoMapperConfiguration.Configure();
Trace.WriteLine("Configuring Windsor...");
_container = new WindsorContainer();
Trace.WriteLine(string.Format("Installing assemblies from directory...{0}",
Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot)));
_container.Install(FromAssembly.InDirectory(
new AssemblyFilter(Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot))));
Trace.WriteLine(string.Format("Setting the default connection limit..."));
ServicePointManager.DefaultConnectionLimit = 12;
}
finally
{
Trace.WriteLine("WorkerRole.Initialize - Complete");
}
}
/// <summary>
/// Starts all of the jobs.
/// </summary>
private void StartJobs()
{
Trace.WriteLine("WorkerRole.StartJobs()");
try
{
foreach (IJob job in _jobs)
{
job.Start();
}
}
finally
{
Trace.WriteLine("WorkerRole.StartJobs - Complete");
}
}
public override void OnStop()
{
Trace.WriteLine("WorkerRole.OnStop()");
try
{
foreach (IJob job in _jobs)
{
job.Stop();
}
_container.Dispose();
}
finally
{
Trace.WriteLine("WorkerRole.OnStop - Complete");
}
}
#endregion
#region Private util classes
public static class AutoMapperConfiguration
{
public static void Configure()
{
Mapper.Initialize(x => x.AddProfile<ModelProfile>());
}
}
#endregion
}
TraceUtil Code:
public static class TraceUtil
{
public static void TraceException(Exception ex)
{
StringBuilder buffer = new StringBuilder();
while (ex != null)
{
buffer.AppendFormat("{0} : ", ex.GetType());
buffer.AppendLine(ex.Message);
buffer.AppendLine(ex.StackTrace);
ex = ex.InnerException;
}
Trace.TraceError(buffer.ToString());
}
}
Config:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
...
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
name="AzureDiagnostics">
<filter type="" />
</add>
</listeners>
</trace>
</system.diagnostics>
</configuration>
Once the worker has started, if I look in the WADLogsTable, all I see is “WorkerRole.OnStart()” and nothing else!
Any ideas on what the problem could be or how to troubleshoot this would be appreciated.
Update: If I stop the role, I don’t see any of the debug statements from the OnStop() method either.
Update: I must have something configured incorrectly with my diagnostics. I thought I was seeing my debug come out correctly when debugging locally, but it turns out I’m not. I see everything in the output window, but I don’t see everything in the storage table. I am seeing the following entries in development:
WorkerRole.OnStart()
WorkerRole.Initialize()
Configuring AutoMapper...
I realize that the trace output is only periodically uploaded, but I’ve waited 5 minutes or so, so I think this should be long enough, since I have it set to 1 minute.
Update: As suggested by @kwill in the comments section I have tried adding a file trace listener as follows:
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
name="AzureDiagnostics">
</add>
<add name="File" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:\TextWriterOutput.log" />
</listeners>
</trace>
</system.diagnostics>
This works fine in my development environment and seems more reliable as well as I get all of the debug out that I would expect. When I deploy it to staging however, the TextWriterOutput.log file isn’t even created!
I really need a reliable way of getting debug out of my worker role so that I can troubleshoot the ultimate problem, which is that my jobs aren’t working – at this point I still have no idea what they are even trying to do as I can’t get any debug out!
Update: I am pretty sure that the missing dll idea suggested by most people is not the problem. To hopefully prove this, I have overridden the run method as shown below and I am seeing the “Heartbeat…” debug come out. It seems to me that either the diagnostics functionality or at least the way I have it configured is unreliable, which is preventing me from investigating why on earth my jobs aren’t running.
public override void Run()
{
Trace.WriteLine("LearningMiles.JobProcessor.WorkerRole.Run()", "Information");
try
{
while (true)
{
Thread.Sleep(10000);
Trace.WriteLine("Heartbeat...", "Verbose");
}
}
catch (Exception ex)
{
TraceUtil.TraceException(ex);
throw;
}
finally
{
Trace.WriteLine("LearningMiles.JobProcessor.WorkerRole.Run() - Complete", "Information");
}
}
Update: I have now cross posted this problem on the Windows Azure MSDN forum.
Update: As suggested in the comments, I have now tried removing all ‘useful’ code. In development this resulted in all of the debug being output. I then tried just removing the call to AutomapperConfiguration.Configure() since previously I was seeing nothing come out after that call. This resulted in some of the trace statements not coming out again. Importantly however, I was seeing the trace statements that I have put in the “jobs”. Since it’s the jobs not running that I ultimately want to resolve, I deployed that version of the code to staging, but there I just see the OnStart() trace and the “heartbeat” trace. I don’t think this really helps, but maybe it will give someone some ideas.
Thanks to an answer on the MSDN forum, I have been able to troubleshoot and resolve my problem.
The reason why my jobs were not being executed was due to the line below:
The role root on staging is E:. Path.Combine() has an obscure implementation which you can read more about in this SO answer. What this meant was that Castle was searching for assemblies in E:approot rather than E:\approot as I expected. I am now constructing the approot path with the method below:
That has resolved my main issue and I am now seeing the jobs execute as expected.
I was only able to troubleshoot this issue by running the worker role in an elevated execution context so that my trace data could be written to a file. I still don’t know why, and would like to hear of any ideas why, the trace statements were not being transferred to storage correctly.