Question

Assume a table named ‘log’, there are huge records in it.

The application usually retrieves data by simple SQL:

SELECT * 
FROM log 
WHERE logLevel=2 AND (creationData BETWEEN ? AND ?)

logLevel and creationData have indexes, but the number of records makes it take longer to retrieve data.

How do we fix this?

Answer 1

Look at your execution plan / “EXPLAIN PLAN” result – if you are retrieving large amounts of data then there is very little that you can do to improve performance – you could try changing your SELECT statement to only include columns you are interested in, however it won’t change the number of logical reads that you are doing and so I suspect it will only have a neglible effect on performance.

If you are only retrieving small numbers of records then an index of LogLevel and an index on CreationDate should do the trick.

UPDATE: SQL server is mostly geared around querying small subsets of massive databases (e.g. returning a single customer record out of a database of millions). Its not really geared up for returning truly large data sets. If the amount of data that you are returning is genuinely large then there is only a certain amount that you will be able to do and so I’d have to ask:

What is it that you are actually trying to achieve?

• If you are displaying log messages to a user, then they are only going to be interested in a small subset at a time, and so you might also want to look into efficient methods of paging SQL data – if you are only returning even say 500 or so records at a time it should still be very fast.

• If you are trying to do some sort of statistical analysis then you might want to replicate your data into a data store more suited to statistical analysis. (Not sure what however, that isn’t my area of expertise)