Question

I have a simple query which returns 25,026 rows:

MySqlCommand cmd = new MySqlCommand("SELECT ID FROM People", DB);
MySqlDataReader reader = cmd.ExecuteReader();

(ID is an int.) If I just do this:

int i = 0;
while (reader.Read()) i++;

i will equal 25026. However, I need to do some processing on each ID in my loop; each iteration ends up taking somewhere in the hundreds of milliseconds.

int i = 0;
MySqlCommand updater = new MySqlCommand("INSERT INTO OtherTable (...)", anotherConnection);
updater.Prepare();
while (reader.Read()) {
     int id = reader.getInt32(0);
     // do stuff, then
     updater.ExecuteNonQuery();
     i++;
}

However, after about 4:15 of processing, reader.Read() simply returns false. In most of my test runs, i equaled 14896, but it also sometimes stops at 11920. The DataReader quitting after the same number of records is suspicious, and the times it stops after a different number of rows seems even stranger.

Why is reader.Read() returning false when there’s definitely more rows? There are no exceptions being thrown – not even first chance exceptions.

---

Update: I mentioned in my response to Shaun’s answer that I was becoming convinced that MySqlDataReader.Read() is swallowing an exception, so I downloaded Connector/Net’s source code (bzr branch lp:connectornet/6.2 C:/local/path) and added the project to my solution. Sure enough, after 6:15 of processing, an exception!

The call to resultSet.NextRow() throws a MySqlException with a message of “Reading from the stream has failed.” The InnerException is a SocketException:

{ Message: "An existing connection was forcibly closed by the remote host",
  ErrorCode: 10054,
  SocketErrorCode: ConnectionReset }

10054 means the TCP socket was aborted with a RST instead of the normal disconnection handshake (FIN, FIN ACK, ACK), which tells me something screwy is happening to the network connection.

In my.ini, I cranked interactive_timeout and wait_timeout to 1814400 (seconds) to no avail.

So… why is my connection getting torn down after reading for 6:15 (375 sec)?

(Also, why is this exception getting swallowed when I use the official binary? It looks like it should bubble up to my application code.)

Answer 1

Since I’m just reading ints, I ultimately just read the entire resultset into a List<int>, closed the reader, and then did my processing. This is fine for ints since a even a million take up < 100 MB of RAM, but I’m still disappointed that the root issue isn’t resolved – if I were reading more than a single int per row, memory would become a very large problem with a large dataset.