Question

I’ve got a UI which can queue up jobs of radically different types. At present, it does this by storing the job in the appropriate table in a database (Table-per-JobType)

My back-end process then comes along, picks up the job(s), and assuming there’s an available worker (thread), uses it to execute the appropriate method for the job.

in PseudoCode:

While(Runnning) {
    While(Queue1.HasJobs && Workers.IdleCount > 0) {
        FirstIdleWorker.Execute(Queue1Method(Type1Job));
    }

    ...

    While(QueueN.HasJobs && Workers.IdleCount > 0) {
        FirstIdleWorker.Execute(QueueNMethod(TypeNJob));
    }
    //Wait for a job to complete or a polling timeout if queues are empty
}

(It’s not actually that naive but it illustrates the order in which work is processed)

As you can see, this works but it doesn’t take into account what order jobs were added in. This isn’t a deal breaker as jobs are atomic but it is annoying from a UI perspective – eg User 1 Queues up 20 jobs of Type 2, then User 2 Queues up 1,000 jobs of Type 1. User 1 now has to wait for all User 2’s jobs to complete before their (relatively) quick jobs are processed.

Jobs do have a CreatedOn property so determining the order isn’t difficult but how best to implement a combined queue in a strongly-typed way that isn’t spaghetti code?

I’m trying to avoid a “GenericJob” object with a .CreatedOn, .Queue1Id, .Queue2Id, .Queue3Id as this just feels sloppy.

While FIFO is broadly what I’m after, it’s not a strict requirement – I just don’t want items to be bumped forever.

Is there a pattern for this sort of thing? If not, can someone point me at a good tutorial please?

(Incidentally, these are potentially long-running jobs. I’m actually using the TPL behind the scenes to manage workers once the jobs are taken from the Queue but I do still need to manage the queue myself as there are far more jobs than I could load into memory in one go)

Answer 1

Poke me if I’m wrong, I hope this pseudo-code explains the Interface Abstraction okay.

An interface might look like:

enum JobTypes
{
    JobType1 = 0x01,
    JobType2 = 0x02,
    JobType3 = 0x03
}
interface IJob
{
    int ID { get; set; }
    JobTypes JobType { get; set; }
    DateTime Date { get; set; }
    bool Complete { get; set; }
    void Process(List<object> parameters);
}

Each job type processor implements this interface, change and add properties to suit your needs:

class JobType1 : IJob
{
    public int ID { get; set; }
    public JobTypes JobType { get; set; }
    public DateTime Date { get; set; }
    public bool Complete { get; set; }
    public void Process(List<object> parameters)
    {
        throw new NotImplementedException();
    }
}

You could then mix the job types into one list:

List<IJob> joblist = new List<IJob>();

and possibly sort them by date using lambda:

joblist.Sort((a, b) => DateTime.Compare(a.Date, b.Date));

get a list of unprocessed jobs (does not check for those busy processing*)

List<IJob> undone = joblist.Where(job => job.Complete == false) as List<IJob>;