My application is using windows azure and SQL Database(Azure). In near future we will be having traffic for our SQL Database near to 50,000 transactions/min. I am using 5GB Web database.
Now there is a limit of 400 concurrent operations per partition in SQL Database:Reference
What are the possible ways to overcome this limitation? At the moment the best solution i can think of is federation. Which are the other ways? And which one is the best?
EDIT : Transactions will be write only. We are collecting performance data from our client systems every 5 seconds which are sent to our REST API which eventually inserts data to SQL Database. So no UI and caching is out of question.
Using Federations in Windows Azure SQL Database is one option. But I prefer offloading the heavy work to different data store like Table Storage, Blobs or Queues since they were built to handle a heavy load and they partition much easier. Combine this with good caching and you might overcome this limitation with ease.
Imagine your site has a top 10 products list on your home page and you have 100,000 visitors / day. One option would be to query SQL Azure each time, but this could cause a heavy load on SQL Azure. But you could have a worker process running every 24 hour for example and calculating the top 10 products at that time, and save them in Table Storage (you could make a few partitions in that table containing the top 10 per country, per category, …). You could see this as a pre-generated view. Each time you want to show the top 10 products you would query the items from Table Storage (a specific partition in a table) which scales much better. Add some ASP.NET caching there and you’ll have a very reliable system.
That was it for reading data. But I imagine you’ll also be expecting some user input, where a user can create an order, send messages, … There again, if you’re expecting a heavy load SQL Azure might not be your best option to interact with directly (seen the limitations). Using queues between your front end and your back end might be a better solution.
When your users place an order you can write a message to a queue (Storage Queue or Service Bus Queue). A worker will pick up this message, and create a record in the table for that order with the time, product count and maybe even a status (like processing) that you would show in the user’s orders screen. Once you’ve done this you have all the time you need to do finish the order and once you finished it you save the end result in SQL Azure (and you update the order record in table storage). Saving the end result in SQL Azure will still make it possible to use reporting etc…
This will have a big impact on your application. An other solution would be to host SQL Server yourself in a Virtual Machine, but note that this is still in CTP: Provisioning a SQL Server Virtual Machine on Windows Azure