Home/ Questions/Q 612147
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T17:51:42+00:00

I am faced with the choice where to store some reference data (essentially drop

  • 0

I am faced with the choice where to store some reference data (essentially drop down values) for my application. This data will not change (or if it does, I am fine with needing to restart the application), and will be frequently accessed as part of an AJAX autocomplete widget (so there may be several queries against this data by one user filling out one field).

Suppose each record looks something like this:

category
effective_date
expiration_date
field_A
field_B
field_C
field_D

The autocomplete query will need to check the input string against 4 fields in each record and discrete parameters against the category and effective/expiration dates, so if this were a SQL query, it would have a where clause that looks something like:

... WHERE category = ? 
AND effective_date < ?
AND expiration_date > ? 
AND (colA LIKE ? OR colB LIKE ? OR colC LIKE ?)

I feel like this might be a rather inefficient query, but I suppose I don’t know enough about how databases optimize their indexes, etc. I do know that a lot of really smart people work really hard to make database engines really fast at this exact type of thing.

The alternative I see is to store it in my application memory. I could have a list of these records for each category, and then iterate over each record in the category to see if the filter criteria is met. This is definitely O(n), since I need to examine every record in the category.

Has anyone faced a similar choice? Do you have any insight to offer?

EDIT: Thanks for the insight, folks. Sending the entire data set down to the client is not really an option, since the data set is so large (several MB).

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Definitely cache it in memory if it’s not changing during the lifetime of the application. You’re right, you don’t want to be going back to the database for each call, because it’s completely unnecessary.

    There’s can be debate about exactly how much to cache on the server (I tend to cache as little as possible until I really need to), but for information that will not change and will be accessed repeatedly, you should almost always cache that in the Application object.

    Given the number of directions you’re coming at this data (filtering on 6 or more columns), I’m not sure how much more you’ll be able to optimize the information in memory. The first thing I would try is to store it in a list in the Application object, and query it using LINQ-to-objects. Or, if there is one field that is used significantly more than the others, or try using a Dictionary instead of a list. If the performance continues to be a problem, try using storing it in a DataSet and setting indexes on it (but of course you loose some code-simplicity and maintainability this way).

    • 0