Question

I’ve been searching all over the place for streaming a file into MySQL using C, and I can’t find anything. This is pretty easy to do in C++, C#, and many other languages, but I can’t find anything for straight C.

Basically, I have a file, and I want to read that file into a TEXT or BLOB column in my MySQL database. This can be achieved pretty easily by looping through the file and using subsequent CONCAT() calls to append the data to the column. However, I don’t think this is as elegant as a solution, and is probably very error prone.

I’ve looked into the prepared statements using mysql_stmt_init() and all the binds, etc, but it doesn’t seem to accept a FILE pointer to read the data into the database.

It is important to note I am working with very large files that cannot be stored in RAM, so reading the entire file into a temporary variable is out of the question.

Simply put: how can I read a file from disk into a MySQL database using C? And keep in mind, there needs to be some type of buffer (ie, BUFSIZ due to the size of the files). Has anyone achieved this? Is it possible? And I’m looking for a solution that works both with text and binary files.

Answer 1

I don’t like answering my own questions, but I feel the need in case someone else is looking for a solution to this down the road.

Unless I’m missing something, my research and testing has shown me that I have three general options:

1. Decent Solution: use a LOAD DATA INFILE statement to send the file
   • pros: only one statement will ever be needed. Unlike loading the entire file into memory, you can tune the performance of LOAD DATA on both the client and the server to use a given buffer size, and you can make that buffer much smaller, which will give you “better” buffer control without making numerous calls
   • cons: First of all, the file absolutely MUST be in a given format, which can be difficult to do with binary blob files. Also, this takes a fair amount of work to set up, and requires a lot of tuning. By default, the client will try to load the entire file into memory, and use swap-space for the amount of the file that does not fit into memory. It’s very easy to get terrible performance here, and every time you wish to make a change you have to restart the mysql server.
2. Decent Solution: Have a buffer (eg, char buf[BUFSIZ]), and make numerous queries with CONCAT() calls to update the content
   • pros: uses the least amount of memory, and gives the program better control over how much memory is being used
   • cons: takes up A LOT of processing time because you are making numerous mysql calls, and the server has to find the given row, and then append a string to it (which takes time, even with caching)
3. Worst Solution: Try to load the entire file into memory (or as much as possible), and make only one INSERT or UPDATE call to mysql
   • pros: limits the amount of processing performance needed on the client, as only a minimum number of calls (preferably one) will need to be buffered and executed.
   • cons: takes up a TON of memory. If you have numerous clients making these large calls simultaneously, the server will run out of memory quickly, and any performance gains will turn to losses very quickly.

In a perfect world, MySQL would implement a feature which allowed for buffering queries, something akin to buffering a video: you open a MySQL connection, then within that open a ‘query connection’ and stream the data in buffered sets, then close the ‘query connection’

However, this is NOT a perfect world, and there is no such thing in MySQL. this leaves us with the three options shown above. I decided to stick with the second, where I make numerous CONCAT() calls because my current server has plenty of processing time to spare, and I’m very limited on memory in the clients. For my unique situation, trying to beat my head around tuning LOAD DATA INFILE doesn’t make sense. Every application, however, will have to analyze it’s own problem.

I’ll stress none of these are “perfect” for me, but you can only do the best with what you have.

Points to Adam Liss for giving the LOAD DATA INFILE direction.