Question

If I should be approaching this problem through a different method, please suggest so. I am creating an item based collaborative filter. I populate the db with the LinkRating2 class and for each link there are more than a 1000 users that I need to call and collect their ratings to perform calculations which I then use to create another table. So I need to call more than 1000 entities for a given link.

For instance lets say there are over a 1000 users rated ‘link1’ there will be over a 1000 instances of this class for the given link property that I need to call.
How would I complete this example?

class LinkRating2(db.Model):
    user = db.StringProperty()
    link = db.StringProperty()
    rating2 = db.FloatProperty()

query =LinkRating2.all()
link1 = 'link string name'
a = query.filter('link = ', link1)
aa = a.fetch(1000)##how would i get more than 1000 for a given link1 as shown?


##keybased over 1000 in other post example i need method for a subset though not key
class MyModel(db.Expando):
        @classmethod
        def count_all(cls):
            """
            Count *all* of the rows (without maxing out at 1000)
            """
            count = 0
            query = cls.all().order('__key__')
            while count % 1000 == 0:
                current_count = query.count()
                if current_count == 0:
                    break
                count += current_count

                if current_count == 1000:
                    last_key = query.fetch(1, 999)[0].key()
                    query = query.filter('__key__ > ', last_key)

            return count

Answer 1

The 1000-entity fetch limit was removed recently; you can fetch as many as you need, provided you can do so within the time limits. Your entities look like they’ll be fairly small, so you may be able to fetch significantly more than 1000 in a request.