I have a scenario where I am processing a large log file by ingesting it onto a Camel-based ESB. The first stop on this bus is a processor – LogTransformer – that parses log messages out of the log file, one by one (real-time), and creates XML snippets:
<log-record level="INFO" source="MyApp" .../>
Each one of these <log-record/> XML snippets gets put onto a message queue and gets picked up by the bus. Eventually it winds up in a MySQL DB.
I have certain processors that I want to kick off only after the log file is completely processed (all log records have been converted to <log-record/> and enqueued, processed by the bus, and stored to the DB). Things like report generators, BI components, etc. These processors shouldn’t begin until the entire log is enqueued.
What’s the Camel way of automatically indicating to these processors that they can kick off?
The best I can come up with is to have my LogTransformer processor (the part that parses log records, transforms them to <log-record> XML, and enqueues them) create an EOF log record after it finishes processing the log. Something like <log-record eof="true">. It then enqueues this message like normal.
When the last processor in the bus (which persists <log-record> to MySQL) encounters the EOF log record, instead of persisting it, it enqueues it on a queue.
These report generators, BI components, etc. wait for a message on this queue to kick off.
This solution seems convoluted. I’m brand new to ESBs, and Camel, so I figured there would be an EIP/processor that handles this scenario, but I can’t seem to find it. How would a Camel Wizard structure this solution?
Have a look at Splitter (http://camel.apache.org/splitter.html) and Aggregator (http://camel.apache.org/aggregator2.html). The aggregator can use completionSize to determine that it has all of the pieces it needs to continue processing.