Question

In one of my process I have this SQL query that take 10-20% of the total execution time. This SQL query does a filter on my Database, and load a list of PricingGrid object.
So I want to improve these performance.
So far I guessed 2 solutions :

Use a NoSQL solution, AFAIK these are good solutions for improving reading process.

• But the migration seems hard and needs a lot of work (like import the data from sql server to nosql in a regular basis)
• I don’t have any knowledge , I even don’t know which one I should use (the first I’d use is Ravendb because I follow ayende and it’s done by the .net community).
• I might have some stuff to change in my model to make my object ok for a nosql database

Load all my PricingGrid object in memory (in a static IEnumerable)

• This might be a problem when my server won’t have enough memory to load everything
• I might reinvent the wheel (indexes…) invented by the NoSQL providers

I think I’m not the first one wondering this, so what would be the best solution ? Is there any tools that could help me ?

.net 3.5, SQL Server 2005, windows server 2005

Answer 1

Asaf has provided great information in regards to the usage of NoSQL and when it is most appropriate. Given that your main concern was performance, I would tend to agree with his opinion – it would take you much more time and effort to adopt a completely new (and very different) data persistence platform than it would to trick out your SQL Server cluster. That said, my answer is mainly to address the “how” part of your question.

Addressing misunderstandings:

1. Denormalizing Data – You do not need to manually denormalize your existing data. This will be done for you when it is migrated over. More than anything you need to simply think about your data in a different fashion – root aggregates, entity and value types, etc.

2. Concurrency/Transactions – Transactions are possible in both Mongo and Raven, they are simply done in a different fashion. One of the inherent ways Raven does this is by using an ORM-like “unit of work” pattern with its RavenSession objects. Yes, your data validation needs to be done in code, but you already should be doing it there anyway. In my experience this is an over-hyped con.

How:

1. Install Raven or Mongo on a primary server, run it as a service.

2. Create or extend an existing application that uses the database you intend to port. This application needs all the model classes/libraries that your SQL database provides persistence for.

   a. In your “data layer” you likely have a repository class somewhere. Extract an interface form this, and use it to build another repository class for your Raven/Mongo persistence. Both DB’s have plenty good documentation for using their APIs to push/pull/update changes in the document graphs. It’s pretty damn simple.

   b. Load your SQL data into C# objects in memory. Pull back your top-level objects (just the entities) and load their inner collections and related data in memory. Your repository is probably already doing this (ex. when fetching an Order object, ensure not only its properties but associated collections like Items are loaded in memory.

   c. Instantiate your Raven/Mongo repository and push the data to it. Primary entities become “top level documents” or “root aggregates” serialized in JSON, and their collections’ data nested within. Save changes and close the repository. Note: You may break this step down into as many little pieces as your data deems necessary.

3. Once your data is migrated, play around with it and ensure you are satisfied. You may want to modify your application Models a little to adjust the way they are persisted to Raven/Mongo – for instance you may want to make both Orders and Items top-level documents and simply use reference values (much like relationships in RDBMS systems). Watch out here though, as doing so sort-of goes against the principal and performance behind NoSQL as now you have to tap the DB twice to get the Order and the Items.

4. If satisfied, shard/replicate your mongo/raven servers across your remaining available server boxes.

Obviously there are tons of little details I did not explain, but that is the general process, and much of it depends on the applications already consuming the database and may be tricky if more than one app/system talks to it.

Lastly, just to reiterate what Asaf said… learn as much as you can about NoSQL and its best use-cases. It is an amazing tool, but not golden solution for all data persistence. In your case try to really find the bottlenecks in your current solution and see if they are solvable. As one of my systems guys says, “technology for technology’s sake is bullshit”