Question

I need to write a Python script which, at one point, does a Yahoo web search to find and download a bunch of C source files. I’m very new to this and I can’t figure out how to just get started with doing a simple web search… I’ve seen a lot of stuff about BOSS but, from my understanding, this is something you need to pay to use? I am not willing to pay for this.

I’ve used Python YQL to get some RSS results as follows:

import yql
y = yql.Public()
result = y.execute('select * from rss where url="http://www.un.org/apps/news/rss/rss_top.asp"');

for row in result.rows:
   print row.get('title')

And this seems to work, but I can’t figure out how to just do a normal web search (since the search.web table is apparently gone). A basic working example would be much appreciated.

Answer 1

You might want to try it using mechanize, which simulates a browser. If you need to clean out some of the crud in the resulting files, use Beautiful Soup.