Question

Currently, I would like provide this as an option to the user when storing data to the database.

Save the data to a file and use a background thread to read data from the textfile to SQL server.

Flow of my program:
– A stream of data coming from a server constantly (100 per second).
– want to store the data in a textfile and use background thread to copy data from the textfile back to the SQL database constantly as another user option.

Has this been done before?

Cheers.

Answer 1

Your question is indeed a bit confusing.

I’m guessing you mean that:

• 100 rows per second come from a certain source or server (eg. log entries)
• One option for the user is textfile caching: the rows are stored in a textfile and periodically an incremental copy of the contents of the textfile into (an) SQL Server table(s) is performed.
• Another option for the user is direct insert: the data is stored directly in the database as it comes in, with no textfile in between.

Am I right?

If yes, then you should do something in the lines of:

• Create a trigger on an INSERT action to the table
• In that trigger, check which user is inserting. If the user has textfile caching disabled, then the insert can go on. Otherwise, the data is redirected to a textfile (or a caching table)
• Create a stored procedure that checks the caching table or text file for new data, copies the new data into the real table, and deletes the cached data.
• Create an SQL Server Agent job that runs above stored procedure every minute, hour, day…

Since the interface from T-SQL to textfiles is not very flexible, I would recommend using a caching table instead. Why a textfile?

And for that matter, why cache the data before inserting it into the table? Perhaps we can suggest a better solution, if you explain the context of your question.