Question

Edit I now realize the API is simply inadequate and is not even working.
I would like to redirect my question, I want to be able to auto-magically search duckduckgo using their “I’m feeling ducky”. So that I can search for “stackoverflow” for instance and get the main page (“https://stackoverflow.com/“) as my result.

I am using the duckduckgo API. Here

And I found that when using:

r = duckduckgo.query("example")

The results do not reflect a manual search, namely:

for result in r.results:
    print result

Results in:

>>> 
>>> 

Nothing.

And looking for an index in results results in an out of bounds error, since it is empty.

How am I supposed to get results for my search?

It seems the API (according to its documented examples) is supposed to answer questions and give a sort of “I’m feeling ducky” in the form of r.answer.text

But the website is made in such a way that I can not search it and parse results using normal methods.

I would like to know how I am supposed to parse search results with this API or any other method from this site.

Thank you.

Answer 1

If you visit DuckDuck Go API Page, you will find some notes about using the API. The first notes says clearly that:

As this is a Zero-click Info API, most deep queries (non topic names)
will be blank.

An here’s the list of those fields:

Abstract: ""
AbstractText: ""
AbstractSource: ""
AbstractURL: ""
Image: ""
Heading: ""
Answer: ""
Redirect: ""
AnswerType: ""
Definition: ""
DefinitionSource: ""
DefinitionURL: ""
RelatedTopics: [ ]
Results: [ ]
Type: ""

So it might be a pity, but their API just truncates a bunch of results and does not give them to you; possibly to work faster, and seems like nothing can be done except using DuckDuckGo.com.

So, obviously, in that case API is not the way to go.

As for me, I see only one way out left: retrieving raw html from duckduckgo.com and parsing it using, e.g. html5lib (it worth to mention that their html is well-structured).

It also worth to mention that parsing html pages is not the most reliable way to scrap data, because html structure can change, while API usually stays stable until changes are publicly announced.

Here’s and example of how can be such parsing achieved with BeautifulSoup:

from BeautifulSoup import BeautifulSoup
import urllib
import re

site = urllib.urlopen('http://duckduckgo.com/?q=example')
data = site.read()

parsed = BeautifulSoup(data)
topics = parsed.findAll('div', {'id': 'zero_click_topics'})[0]
results = topics.findAll('div', {'class': re.compile('results_*')})

print results[0].text

This script prints:

u'Eixample, an inner suburb of Barcelona with distinctive architecture'

The problem of direct querying on the main page is that it uses JavaScript to produce required results (not related topics), so you can use HTML version to get results only. HTML version has different link:

• http://duckduckgo.com/?q=example # JavaScript version
• http://duckduckgo.com/html/?q=example # HTML-only version

Let’s see what we can get:

site = urllib.urlopen('http://duckduckgo.com/html/?q=example')
data = site.read()
parsed = BeautifulSoup(data)

first_link = parsed.findAll('div', {'class': re.compile('links_main*')})[0].a['href']

The result stored in first_link variable is a link to the first result (not a related search) that search engine outputs:

http://www.iana.org/domains/example

To get all the links you can iterate over found tags (other data except links can be received similar way)

for i in parsed.findAll('div', {'class': re.compile('links_main*')}):
    print i.a['href']

http://www.iana.org/domains/example
Posts by example
https://www.facebook.com/leadingbyexample
http://www.trythisforexample.com/
http://www.myspace.com/leadingbyexample?_escaped_fragment_=

https://en.wikipedia.org/wiki/Example_(musician)
http://www.merriam-webster.com/dictionary/example
...

Note that HTML-only version contains only results, and for related search you must use JavaScript version. (vithout html part in url).