Home/ Questions/Q 698057
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-14T03:13:59+00:00

I am working with a rather large mysql database (several million rows) with a

  • 0

I am working with a rather large mysql database (several million rows) with a column storing blob images. The application attempts to grab a subset of the images and runs some processing algorithms on them. The problem I’m running into is that, due to the rather large dataset that I have, the dataset that my query is returning is too large to store in memory.

For the time being, I have changed the query to not return the images. While iterating over the resultset, I run another select which grabs the individual image that relates to the current record. This works, but the tens of thousands of extra queries have resulted in a performance decrease that is unacceptable.

My next idea is to limit the original query to 10,000 results or so, and then keep querying over spans of 10,000 rows. This seems like the middle of the road compromise between the two approaches. I feel that there is probably a better solution that I am not aware of. Is there another way to only have portions of a gigantic resultset in memory at a time?

Cheers,

Dave McClelland

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    One option is to use a DataReader. It streams the data, but it’s at the expense of keeping an open connection to the database. If you’re iterating over several million rows and performing processing for each one, that may not be desirable.

    I think you’re heading down the right path of grabbing the data in chunks, probably using MySql’s Limit method, correct?

    • 0