Question

Let’s say I have a Job Scheduler which has 4 consumers A, B, C and D. Jobs of type X will have to be routed to Consumer A, type Y to Consumer B and so on. Consumers A, B, C and D are to run as independent applications without any dependency, either locally or remotely.

The consumers take varying times to complete their jobs, which are subsequently routed to the Job Scheduler for aggregation.

Clones of one of the consumers may also be needed to share its eligible jobs. A job should however be processed only once.

Is Content-based router the best solution for this? Mind you, I need the custom job scheduler, because it only has the intelligence to split up a job among the consumers.

Or is there any better way to handle this? I don’t require those features of the broker like automatically switching over to another consumer (load balancing) and such in case of a failure.

Answer 1

I’m not completly sure that I follow you. This sounds like a rather straight forward scenario for asychronous processing.

I’m not sure how you plan to send these jobs to the Camel application, but given you can receive them somewhere you could probably go ahead with a simple content based router.

Given your requirements for the consumers, I would go for JMS queues (using Apache ActiveMQ or similar broker middleware), one queue per job type. This makes it easy to distribute consumers to different machines without really changing the code.

// Central node routes
from("xxx:routeJob")
   .choice()
       .when(header("type").isEqualTo("x"))
           .to("jms:queue:processJobTypeX")
       .when(header("type").isEqualTo("y"))
           .to("jms:queue:processJobTypeY")
       .otherwise()
           .to("jms:queue:processJobTypeZ"); 

 from("jms:queue:aggregateJob")
    .bean(aggregate);

// different camel application (may be duplicated for multiple instances).
from("jms:queue:processJobTypeX")
  .bean(heavyProcessing)
  .to("jms:queue:aggregateJob");

// Yet another camel application
from("jms:queue:processJobTypeY")
  .bean(lightProcessing)
  .to("jms:queue:aggregateJob");

Please revisit your question for a better answer 🙂