Question

This will be my first multi threaded application if it can be developed as so. So I need some help getting started.

I currently have the following JAVA program that works perfectly well

1. Get data from a source database and convert them to objects(POJOs)
2. Get data from a target database and convert them to objects(POJOs).
3. Compare both the source and target objects and if there is a difference, then update the target database with info from source database.

Now my requirement has changed and I need to use 3 different target databases(TargetDB1, TargetDB2, TargetDB3). Each one has a different connection info.

My question is , Can we make this a multi threaded application where in

Thread 1 will get source object

Thread 2 will get TargetDB1 object, compare to the source object(obtained from Thread 1) and update the TargetDB1 in case of differences with the source object

Thread 3 will get TargetDB2 object, compare to the source object(obtained from Thread 1) and update the TargetDB2 in case of differences with the source object

Thread 4 will get TargetDB2 object, compare to the sourceobject (obtained from Thread 1) and update the TargetDB2 in case of differences with the source object

If this can be developed as a multi threaded application, then how to go about it.

Answer 1

I would create a work queue for each of your target databases using some BlockingQueue such as a LinkedBlockingQueue. The producer thread will get the object from the source database and then add it to each of the work queues. Then each of the threads attached to a target database can dequeue from its work queue and update its database at its own speed.

Source database thread might look something like:

// allow only 10 outstanding objects to be in the work queue before it blocks
queue1 = new LinkedBlockingQueue<SourceObject>(10);
new Thread(new TargetDatabaseThread("url-to-database1", queue1)).start();
queue2 = new LinkedBlockingQueue<SourceObject>(10);
new Thread(new TargetDatabaseThread("url-to-database2", queue2)).start();
queue3 = new LinkedBlockingQueue<SourceObject>(10);
new Thread(new TargetDatabaseThread("url-to-database3", queue3)).start();
while (true) {
    SourceObject sourceObj = getFromSourceDatabase();
    // this might block if you set a capacity on your queue and it was full
    queue1.put(sourceObj);
    queue2.put(sourceObj);
    queue3.put(sourceObj);
}

Each of the target database threads then might look like:

public class TargetDatabaseThread implements Runnable {
    private final String jdbcUrl;
    private final BlockingQueue queue;
    private volatile boolean shutdown;
    public TargetDatabaseThread(String jdbcUrl, BlockingQueue queue) {
        this.jdbcUrl = jdbcUrl;
        this.queue = queue;
    }
    public void run() {
        // maybe some initialization, make database connection, etc.
        while (!shutdown) {
            // this would block if nothing is in the queue
            SourceObject sourceObj = queue.take();
            SourceObject targetObj =
                getObjectFromTargetDatabase(sourceObj.getId());
            if (updateTarget(sourceObj, targetObj)) {
               updateMyTargetObjectInDatabase(targetObj);
            }
        }
    }
}

One problem with this model would be if the target database threads need to update the source object in any manner. If this is the case then you would have to synchronize the object.

Note that blocking queues can be set with a capacity. This would be useful in case it was easier for the producer to read from the source database than it was for the target database writers to update their databases. You wouldn’t want the queue to fill memory because the target databases are slow.