I am prepping up for the open graph release.
One of the features from open graph is to crawl the site’s meta data and pull relevant info for the timeline.
My sitemap.xml runs into gigs and is protected by rate limiting and ip whitlisting for popular crawlers like Googlebot & Slurp.
Can someone pass me the robot names and ips that will be crawling for facebook? This is not just for sitemap.xml, but for general rate limit(whole site) white listing too.
I am prepping up for the open graph release. One of the features from
Share
The current user agent is:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)Facebook publishes their IP range here.