Home/ Questions/Q 8877497
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-14T19:28:56+00:00

here’s my problem. I want to make a collection in mongodb where I have

  • 0

here’s my problem. I want to make a collection in mongodb where I have a word and the number of the times it occurs. I’m doing it in python and it’s extrememly slow. It’s most probably because for every word that i have, I check if it is already in the database (using
*find_one*)and if yes, get its frequency, increment it and store it back (using update) Of course, when the word is not there, I append it to a list and do bulk insert periodically.

Is there a better way of doing this? The number of words is huge (different languages possible). Is mongoDB the right thing to use in the first place? I chose mongoDB because it was very easy to install to and I picked up the tutorial in 10 min…

edit – added the code as well. When I say large, I mean a file that is some 4 GB large with words in them…

insertlist = []

def copy_to_db(word):
    global insertlist

    wordCollection = db['words']
    occurrence = wordCollection.find_one({'word' : word})
    if occurrence:
            n = occurrence['number']
            n = n + 1
            wordCollection.update({'word' : word}, {'$set' : {'number' : n}})
    else:
            insertlist.append({'word' : word, 'number' : 1})
            #wordCollection.insert({'word' : word, 'number' : 1})

    if len(insertlist) >= 5000:
            print("insert triggered ... ")
            wordCollection.insert(insertlist)
            insertlist = []

i call this func. for every word.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Sounds like you could use upserts. If you use upserts you don’t need to do that fetch/save cycle.

    I’m not sure how this is done in python driver, but in JavaScript it would look something like:

    db.words.update({"_id": "the_word" }, {"$inc": {"frequency": 1}}, true)

    MongoDB creates an index for the _id field automatically. If you are not using the _id field for your word, then creating an index for your key would most probably help a lot.

    Edit: some more ideas for you

    As there is a lot of data, you could use the _id field for your word. This way you wouldn’t need to create another index and the updates would be slightly faster as only one index needs to be updated while inserting new documents. This is in case inserting speed is the bottleneck.

    While taking advantage of batch inserts is generally a good idea when inserting a lot of data, I’m not sure if it helps too much on this case. This depends on your data. If the ratio of unique words is high, then batch inserts might be handy. But if the same words are used over and over again (which I guess is the case with most languages), then batch inserts might not be of too much help.

    Also, it looks like you have a problem in your batch insert. Think about if you encounter a words for the first time. It is inserted to your insertlist. Now, if this same word is encountered again while the previous batch is not inserted, the number attribute for this word would be 1 which would be incorrect.

    Are you sure the db is the bottleneck? Have you already made sure that the is no other poorly performing code? But anyway I guess, inserting 4GB of data will take a while in any case.

    • 0