Question

What is an example of data that is “predistilled or aggregated in runtime”? (And why isn’t MongoDB very good with it?)

This is a quote from the MongoDB docs:

Traditional Business Intelligence. Data warehouses are more suited to new, problem-specific BI databases. However note that MongoDB can work very well for several reporting and analytics problems where data is pre-distilled or aggregated in runtime — but classic, nightly batch load business intelligence, while possible, is not necessarily a sweet spot.

Answer 1

Let’s take something simple like counting clicks. There are a few ways to report on clicks.

1. Store the clicks in a single place. (file, database table, collection) When somebody wants stats, you run a query on that table and aggregate the results. Of course, this doesn’t scale very well, so typically you use…
2. Batch jobs. Store your clicks as in #1, but only summarize them every 5 minutes or so. When people want to query the summary table. Note that “clicks” may have millions of rows, but “summary” may only have a few thousand rows, so it’s much quicker to query.
3. Count the clicks in real-time. Every time there’s a click you increment a counter somewhere. Typically this means incrementing the “summary” table(s).

Now most big systems use #2. There are several systems that are very good for this specifically (see Hadoop).

#3 is difficult to do with SQL databases (like MySQL), because there’s a lot of disk locking happening. However, MongoDB isn’t constantly locking the disk and tends to have much better write throughput.

So MongoDB ends up being very good for such “real-time counters”. This is what they mean by predistilled or aggregated in runtime.

But if MongoDB has great write throughput, shouldn’t it be good at doing batch jobs?

In theory, this may be true and MongoDB does support Map/Reduce. However, MongoDB’s Map/Reduce is currently quite slow and not on par with other Map/Reduce engines like Hadoop. On top of that, the Business Intelligence (BI) field is filled with many other tools that are very specific and likely better-suited than MongoDB.