Question

I am using an API which has a limit on requests. The API response is in XML and typically it is around 8 kB big.To say short: I want to eliminate to exactly send the same requests twice to the API.

There can be over 2 million requests with their appropriate answers. So when I get that amount of XML responses copied to records in MySQL, will it respond fast enough if I query it?

Table structure:

• id int PK Auto Increment
• request_parameters TEXT UNIQUE
• response TEXT

I found MySQL Query Cache, but I think its only handy for much requested requests. The other will maybe slow down?
I also found Varnish, but it seems to me more a HTML/code cache.

Answer 1

I assume:

• you only need to fetch a response from a given request_parameters (...WHERE request_parameters = 'some string')
• request_parameters are not too long (under 1000 characters)

I advise:

1. Drop your ID column. Although some people recommend always using an integer as a primary key, I understand you are not trying to build a relational database. You just need one table for storage.

2. Change your request_parameters column types to VARCHAR(x), “x” being the largest expected size of a parameter string.

3. Make request_parameters the primary key.

4. As advised by PLB, store the serialized version of your XML responses.

2 millions rows of such data will require several Gb, plus some space for the index. If you have a hell lot of available memory on your system, and if you can afford loosing all your data when you restart MySQL, then you might want to go for a try with MEMORY table.