Home/ Questions/Q 6909281
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T08:40:10+00:00

I am developing crawler using anemone gem (Ruby- 1.8.7 and Rails 3.1.1). How should

  • 0

I am developing crawler using anemone gem (Ruby- 1.8.7 and Rails 3.1.1). How should I skip web-pages with extensions pdf, doc, zip, etc. from crawling/downloading.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team
    ext = %w(flv swf png jpg gif asx zip rar tar 7z gz jar js css dtd xsd ico raw mp3 mp4 wav wmv ape aac ac3 wma aiff mpg mpeg avi mov ogg mkv mka asx asf mp2 m1v m3u f4v pdf doc xls ppt pps bin exe rss xml)

Anemone.crawl(url) do |anemone|

    anemone.skip_links_like /\.#{ext.join('|')}$/

    ...

end
    • 0