Home/ Questions/Q 1018777
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-16T10:54:34+00:00

I have around 2 million strings with different lengths that I need to compress

  • 0

I have around 2 million strings with different lengths that I need to compress and put into MongoDb GridFS as files.

The strings are currently stored in MS SQL TEXT field of a table. I wrote a sample app to read each row, compress it and store it as a GridFS file.

There is one reader and a thread pool of 50 threads storing the results. It works but it is very slow (100 records per second on average).

I was wondering if there is any way for faster import into GridFS?

I’m using MongoDb 1.6 on Windows with MongoCSharp driver in C# and .NET.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I think I found the issue inside MongoDb CSharp driver by profiling it while running a very simple app that puts 1000 strings into 1000 GridFS files.

    It turns out that 97% of the time is spent on checking if a file with the same filename exists in the collection. I added an index on the filename field and it’s now blazing fast!

    The question for me is if the driver needs to keep the filename unique and does a check, why doesn’t it add a unique index to it if that’s missing? What’s the reason behind that?

    • 0