Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have a site with around 100,000 unique pages.
(1) How do I create a Sitemap for all these links? Should I just list them flat in a large sitemap protocol compatible file?
(2) Need to implement this on Google App Engine where there is a 1000 item query limit, and all my individual site URLs are stored as separate entries. How do I solve this problem?
Site Maps must be no larger than 10MB and list no more than 50,000 URLs, so you’re going to need to break it up somehow.
You’re going to need some kind of sharding strategy. I don’t know what your data looks like, so for now let’s say every time you create a page entity, you assign it a random integer between 1 and 500.
Next, create a Sitemap index, and spit out a sitemap link for each of your index values:
Finally, on your sitemap page, query for pages and filter for your random index. If you have 100,000 pages this will give you about 200 URLs per sitemap.
A slightly different strategy here would be to give each page an auto-incrementing numeric ID. To do so, you need a counter object that is transactionally locked and incremented each time a new page is created. The downside of this is that you can’t parallelize creation of new page entities. The upside is that you would have a bit more control over how your pages are laid out, as your first sitemap could be pages 1-1000, and so on.