Question

I am about 70% of the way through developing a web application which contains what is essentially a largeish datatable of around 50,000 rows.

The app itself is a filtering app providing various different ways of filtering this table such as range filtering by number, drag and drop filtering that ultimately performs regexp filtering, live text searching and i could go on and on.

Due to this I coded my MySQL queries in a modular fashion so that the actual query itself is put together dynamically dependant on the type of filtering happening.

At the moment each filtering action (in total) takes between 250-350ms on average. For example:-

The user grabs one end of a visual slider, drags it inwards, when he/she lets go a range filtering query is dynamically put together by my PHP code and the results are returned as a JSON response. The total time from the user letting go of the slider until the user has recieved all data and the table is redrawn is between 250-350ms on average.

I am concerned with scaleability further down the line as users can be expected to perform a huge number of the filtering actions in a short space of time in order to retrieve the data they are looking for.

I have toyed with trying to do some fancy cache expiry work with memcached but couldn’t get it to play ball correctly with my dynamically generated queries. Although everything would cache correctly I was having trouble expiring the cache when the query changes and keeping the data relevent. I am however extremely inexperienced with memcached. My first few attempts have led me to believe that memcached isn’t the right tool for this job (due to the highly dynamic nature of the queries. Although this app could ultimately see very high concurrent usage.

So… My question really is, are there any caching mechanisms/layers that I can add to this sort of application that would reduce hits on the server? Bearing in mind the dynamic queries.

Or… If memcached is the best tool for the job, and I am missing a piece of the puzzle with my early attempts, can you provide some information or guidance on using memcached with an application of this sort?

Huge thanks to all who respond.

EDIT: I should mention that the database is MySQL. The siite itself is running on Apache with an nginx proxy. But this question is related purely to speeding up and reducing the database hits, of which there are many.

I should also add that the quoted 250-350ms roundtrip time is fully remote. As in from a remote computer accessing the website. The time includes DNS lookup, Data retrieval etc.

Answer 1

If I understand your question correctly, you’re essentially asking for a way to reduce the number of queries against the database eventhough there will be very few exactly the same queries.

You essentially have three choices:

1. Live with having a large amount of queries against your database, optimise the database with appropriate indexes and normalise the data as far as you can. Make sure to avoid normal performance pitfalls in your query building (lots of ORs in ON-clauses or WHERE-clauses for instance). Provide views for mashup queries, etc.
2. Cache the generic queries in memcached or similar, that is, without some or all filters. And apply the filters in the application layer.
3. Implement a search index server, like SOLR.

I would recommend you do the first though. A roundtrip time of 250~300 ms sounds a bit high even for complex queries and it sounds like you have a lot to gain by just improving what you already have at this stage.
For much higher workloads, I’d suggest solution number 3, it will help you achieve what you are trying to do while being a champ at handling lots of different queries.