Question

I am looking for a way to search in an efficient way for data in a huge multi-dimensional matrix.

My application contains data that is characterized by multiple dimensions. Imagine keeping data about all sales in a company (my application is totally different, but this is just to demonstrate the problem). Every sale is characterized by:

• the product that is being sold
• the customer that bought the product
• the day on which it has been sold
• the employee that sold the product
• the payment method
• the quantity sold

I have millions of sales, done on thousands of products, by hundreds of employees, on lots of days.

I need a fast way to calculate e.g.:

• the total quantity sold by an employee on a certain day
• the total quantity bought by a customer
• the total quantity of a product paid by credit card
• …

I need to store the data in the most detailed way, and I could use a map where the key is the sum of all dimensions, like this:

class Combination
   {
   Product *product;
   Customer *customer;
   Day *day;
   Employee *employee;
   Payment *payment;
   };

std::map<Combination,quantity> data;

But since I don’t know beforehand which queries are performed, I need multiple combination classes (where the data members are in different order) or maps with different comparison functions (using a different sequence to sort on).

Possibly, the problem could be simplified by giving each product, customer, … a number instead of a pointer to it, but even then I end up with lots of memory.

Are there any data structures that could help in handling this kind of efficient searches?

EDIT:

Just to clarify some things: On disk my data is stored in a database, so I’m not looking for ways to change this.

The problem is that to perform my complex mathematical calculations, I have all this data in memory, and I need an efficient way to search this data in memory.

Could an in-memory database help? Maybe, but I fear that an in-memory database might have a serious impact on memory consumption and on performance, so I’m looking for better alternatives.

EDIT (2):

Some more clarifications: my application will perform simulations on the data, and in the end the user is free to save this data or not into my database. So the data itself changes the whole time. While performing these simulations, and the data changes, I need to query the data as explained before.

So again, simply querying the database is not an option. I really need (complex?) in-memory data structures.

Answer 1

EDIT: to replace earlier answer.

Can you imagine you have any other possible choice besides running qsort( ) on that giant array of structs? There’s just no other way that I can see. Maybe you can sort it just once at time zero and keep it sorted as you do dynamic insertions/deletions of entries.