Question

I’m querying Wikipedia using the following code, but I always get an error (403 forbidden). When I type the exact same url in my browser, however, it works. I’ve been using the same code before to query other web apis, so I am not sure what’s causing the trouble.

    private static string query(string text)
    {
        text = text.Replace(" ", "%20");

        string url = "http://en.wikipedia.org/w/api.php?action=opensearch&search=" + text + "&format=json&callback=spellcheck";

        WebClient client = new WebClient();
        client.Headers.Add("User-Agent", "whatever");  // <-- this line was missing

        try
        {
            string response = client.DownloadString(url);
            return response; 
        }
        catch(Exception e)
        {
            Console.WriteLine(e.Message);
            return null; 
        }   
    }

Answer 1

Try setting the user agent header to something that matches your browser. If this doesn’t work, fire up Fiddler, take a peek at your browser headers and copy them to your web request.

http://msdn.microsoft.com/en-us/library/system.net.webclient.headers.aspx

EDIT

The advice I gave was generic. Please observe the policies of the website you are downloading from, as spoofing a browser user-agent may contravene policy or be considered malicious by default:

http://meta.wikimedia.org/wiki/User-Agent_policy :

Do not copy a browser’s user agent for your bot, as bot-like behavior with a browser’s user agent will be assumed malicious.