Question

I have a database in SQL Server 2008 R2 that has millions of files stored in it as varbinary blobs. I set up a process last week to do the following:

1. Use some Entity Framework code to get the row (i.e. entity) that has the blob.
2. Copy the blob stream to an object store.
3. Update entity in the database with the new Object ID from the store.

I have 30 threads doing this constantly and the process will still take several days. A couple days ago the database log file filled up and I’m sure it had to be caused by this process. I decided point-in-time backups are not critical for this database and set the database to use the Simple recovery model. Then yesterday I got errors from my process again that said the log file filled up! How is this possible in simple mode?! Any idea what I can do to stop this from happening? When I monitor the log file SQL Server lets it get to 7% full and then truncates it down to zero so I’m super confused. It seems as though the log file starts to grow unchecked when the full backup begins…

Answer 1

Even in Simple Recovery mode, SQL Server will populate the log file in order to maintain ACID compliance and permit rollbacks (or “roll-forwards” in certain-disaster recovery scenarios). So you need to have enough room in there to at least handle the transactions that it may receive at one time and if not it will either autogrow or present an error.

Now as a practical matter you have a couple of options. The simplest is to just give it sufficient space to handle the transactions. Another is to break it up into fewer transactions since in Simple Recovery it will permit that space to be reused once the transaction is fully committed and finalized.

Edit for clarity and responding to comments: SQL Server will place almost all actions (there are some exceptions, but they aren’t material here) in an implicit transactions even if an explicit one is not called. So, if you say execute a command that inserts a million rows it will have an implicit transaction for that insert and even in simple recovery mode all of those million rows will impact the transaction log until the transaction is fully committed and complete. If you want it to release the space faster, rewrite your code so that instead of one statement inserting a million rows you have ten statements inserting a hundred thousand each.

It wasn’t clear from your question that it was only growing during the full backups, but yes the backup process does affect its ability to release space in the log while it is occurring. That is because a full backup also includes the data from the log file so the Server needs to make sure that information is available during the backup process. There is a related discussion at In Recovery.

I would generally be extremely reluctant to postpone a scheduled backup, but it may help in this particular scenario. It also sounds like breaking your transactions up into smaller pieces may ultimately be the way to go though.