I have a problem accessing the Project Gutenberg Library…
I am using Python 2.7.3.
I can access the NLTK library and work with python, but when attempting to access raw text, it doesn’t allow me to.
The text I was accessing is Crime and Punishment, it’s len(raw) should equal 1176831, but instead gives me a len(raw) of 288.
Here is the code that I used:
>>> from __future__ import division
>>> import nltk, re, pprint
>>> from urllib import urlopen
>>> url = "http://www.gutenberg.org/files/2554/2554.txt"
>>> raw = urlopen(url).read()
>>> type(raw)
<type 'str'>
>>> len(raw)
288
>>> raw
'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don\'t have permission to access /files/2554/2554.txt\non this server.</p>\n<hr>\n<address>Apache Server at www.gutenberg.org Port 80</address>\n</body></html>\n'
>>>
The reason for the HTTP 403 response can be found here. Basically the site is “for human (non-automated) users only. Any perceived use of automated tools to access our web site will result in a temporary or permanent block of your IP address or subnet.”
Your code “should work”, but the website is determining you are accessing the site through code and not a browser. That is all I will say. 🙂