Lets say we have Twitter, and every profile needs to get indexed in search engines, how does Twitter handle their sitemap? Is there something like “regex” sitemap for domain or do they re-generate a sitemap for each user?
How does this work, for pages that you don’t know, so dynamic pages? Look at Wikipedia for example, how do they make sure everything is indexed by Search Engines?
Most likely, they don’t bother to do a sitemap.
For highly dynamic sites, a sitemap will not help that much. Google will index only some amount, and if everything canges before Google considers to revisit it, you don’t gain much.
For slowly changing sites this is different. The sitemaps tells Google on the one hand, which sites exist that it maybe has not yet visited at all, and (more importantly), which site have not changed and thus do not need to be revisited.
But the
sitemap.xmlmechanism just does not scale up to huge and highly dynamic sites such as twitter.