Question

I have a MongoDB Collection called Questions which holds JSON Objects that look like this:

{
    "_id"      : "49902cde5162504500b45c2c", 
    "question" : "Who are you?",
    "category" : "Personal"
}

and I’m trying to select some number, say 1000, of these questions at a time. Once I process those 1000 entries, I want to procede to select the next 1000 entries, and so on.

Is there an easy way to do this sort of action using MongoDB? I’m sort of new to the technology. If not, does anybody have a good way of doing this differently? I’m using the Java driver if that makes a difference.

Thanks,
Chris Covert

Answer 1

An alternate approach is to:

1. Set Last Processed Id = 0

2. Select from questions where id > [last processed id] order by id limit 1000

3. Process these rows
4. Set Last Processed Id = [last id processed from the current 1000 results]
5. Jump back to Step 2

This will be more efficient than selecting everything and skip()’ing past the result set.

Alternatively add a ‘processed’ flag to your records so you can tell apart which records need processing and then query where processed = 0.