Question

The code:

Channel.all.paginate(:page => 3, :per_page => 25)

Say I have a table with 400,000 records, does the above code select all 400,000 records then get the current 25 I need or does it only query for the 25 I need.

If it queries all 400,000 records is there a better optimized way to paginate large datasets using rails?

Answer 1

Mongo Mapper (which I assume your using because of the syntax of your query) is implementing this using the limit and skip expressions.

Basically it would run a query where it skips over a number of Channels and then retrieves the amount specified by the limit (the number you are getting per page).

For example: If you were on page 3 and have 25 per page, the query that mongo mapper runs looks like this:

db.channels.find().skip((page - 1) * per_page).limit(per_page)

Which translates to:

db.channels.find().skip(2 * 25).limit(25)

To return results, mongo has to skip over (page - 1) * per_page number of results which can be costly if the page number is high. Lets say that expression evaluates to 1000, then it would have to run the query, skip over 1000 documents and get the next 25 documents (the limit). MongoDB would essentially be doing a table scan over those documents.

To avoid that you can do range based paging which provides better use of indexes but does not allow you to easily jump to a specific page.

If the Channel model has a date field for example, range based paging would, instead of using skip, use $gte and limit. You would take the date of last document on x page and get the next page’s results by querying for documents with date $gte of previous page’s final document. If you do that you could get dupes though, so it might make sense to use a different criteria.

In practice, don’t worry about it unless you have a really high number of pages.

Cheers and good luck!