Question

Pretty much that is the question. Is there a way that is more efficient than the standart sitemap.xml to [add/force recrawl/remove] i.e. manage your website’s index entries in google?

I remember a few years ago I was reading an article of an unknown blogger that was saying that when he write news in his website, the url entry of the news will appear immediately in google’s search result. I think he was mentioning about something special. I don’t remember exactly what.. . some automatic re-crawling system that is offered by google themselves? However, I’m not sure about it. So I ask, do you think that I am blundering myself and there is NO OTHER way to manage index content besides sitemap.xml ? I just need to be sure about this.

Thank you.

Answer 1

I don’t think you will find that magical “silver bullet” answer you’re looking for, but here’s some additional information and tips that may help:

• Depth of crawl and rate of crawl is directly influenced by PageRank (one of the few things it does influence). So increasing your site’s homepage and internal pages back-link count and quality will assist you.
• QDF – this Google algorithm factor, “Query Deserves Freshness”, does have a real impact and is one of the core reasons behind the Google Caffeine infrastructure project to allow much faster finding of fresh content. This is one of the main reasons that blogs and sites like SE do well – because the content is “fresh” and matches the query.
• XML sitemaps do help with indexation, but they won’t result in better ranking. Use them to assist search bots to find content that is deep in your architecture.
• Pinging, especially by blogs, to services that monitor site changes like ping-o-matic, can really assist in pushing notification of your new content – this can also ensure the search engines become immediately aware of it.
• Crawl Budget – be mindful of wasting a search engine’s time on parts of your site that don’t change or don’t deserve a place in the index – using robots.txt and the robots meta tags can herd the search bots to different parts of your site (use with caution so as to not remove high value content).

Many of these topics are covered online, but there are other intrinsic things like navigational structure, internal linking, site architecture etc that also contribute just as much as any “trick” or “device”.