I have a high performance system (well I think, but not yet there 100% ) written entirely in C# and I think I have made some big architectural mistakes while designing. The reason is that it is not easily scalable.
Though it currently works pretty well, I want to make sure it is horizontally scalable for the increase of volumes which I expect might happen after a few months.
This system has a high number of concurrent connections of data coming into the system which will get into the database eventually after processing. We currently get about 300 records/connections per minute.
The system is architected like this.
- The whole system is hosted on a win 2003 server 8 GB RAM/4 vCPU infra in amazon
- C# Socket server which gets the data and puts into a MSMQ
- A processor for data and insert to sql server 2008 database table. One of the main table has about 3 GB data even after clearing of periodical data. This has the right indexes and currently reports are reasonably faster enough even to a remote location.
- This processed data is then posted to MQ which is then processed for rules which will generate certain alerts
- There are some other associated programs apart from the above
Now the main worry is about scalability of the processor in Step (3) and scalability of Sql server 2008. As the size of concurrent connections increase along with the sql server data, it will make my life tougher.
I have come up with 2 alternatives. One of them is a major replacement of backend processors considering the fact that the current system in entirely built on Microsoft technologies.
For all the options, for the main largest table use a postgresql/pgpool III load balanced(stream replicated) solution for storage. The other tables & schema will still remain in sql 2008. This gives me a cost effective solution of database storage.
Option 1:
– Replace MSMQ with JBOSS & HornetQ
– Put the data processor in step 3 to a container managed “message driven bean” in JBOSS ejb container which will give me options for load balancing and clustering.
– This option will need me to move major part of my solution to unix/linux (am considering fedora)
Option 2:
– Replace MSMQ with Queues of ActiveMQ (Clustered and load balanced)
– Write a Java app which will handle the queue messages and take care of database persistence.
This option will allow me to increase the number of linux servers with the activemq cluster instance and a new instance of the java app.
Option 3:
– Replace MSMQ with Queues of ActiveMQ (Clustered and load balanced)
– Use just the current data processor (with some small changes to push data to postgresql)
This option will force me to remain with Windows
Please note that the system is a real-time system. It is enough if the system is 99% fault proof. This is not a trading system, so I can afford for a little amount of data loss.
Don’t know if I have explained what I want clearly. But I welcome any questions, as they will definitely help me explain it much better.
Please give your valuable suggestions in making the right choice for a long term solution. I am actually against option 3 myself, but don’t want to make a mistake again by leaving it out of the list.
Muthu
Added for clarification:
Apologies for not being clear.
1. The question is actually about scalability of the architecture. Especially horizontal scalability.
2. The current average load is about 300 per minute and this may not exactly be spread inside a single minute.
3. The load might scale to 10 times more easily in the next 8-12 months.
The problem is we were selling about 50 devices in a month and now the sales team is ramping up too fast. I believe this might double soon.
Sql server had about 8 GB of data and we restricted the amount of storage per device and this has helped in reducing the size. Currently the biggest table is partitioned as 1 partition per 200 device and the queries are reasonable. But I can see a bottleneck on the Sql side with the scalability.
So even if the Sql server is put on another server, there is going to be a limit to the amount of simultaneous updates I can do on sql server. I can’t see a horizontal scalability option with load balancing for Sql server (though it supports a High availability option with clustering). did I mis-understand MS Sql in load balancing?
Five updates per second per connection isn’t much depending on the number of connections. You didn’t say how many connections you have, expect to have.
In Java, what I would do in your situation (and I imagine would be just as easy in any technology) is to use batches of data.
The performance issues with messaging and databases is often concerning the rate of messages/transactions you perform. I would have a tasks/thread which takes all the messages pending and roll these into a batch, one MQ message, one transaction to the database. The elegance of this solution is that the slower your MQ messaging, the larger the batches and more efficiently it deals with each connection message. The remaining question is a only, can the messaging/database handle the bandwidth of data.