Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I’ve a project of natural language processing but for that i need to crawl many web articles from some sources like Yahoo news, Google news or blogs…
I’m a java developper (so i’d rather use java tools). I guess i can parse each source website on my own and extract the articles with HttpClient / XPath but i’m a bit lazy 🙂 is there a way so that i won’t have to make a parser per source?
(I’m not only interested by new articles but articles from 2000 to now too)
The hardest part of NLP is getting data you can use. Everything else is just math.
It may be hard to find a large collection of news articles other than on each news source’s website because of all the copyright issues involved. If you don’t need recent news, your best bet is probably to look at the Linguistic Data Consortium’s English Gigaword corpus; if you are at a university, there may already be an existing relationship for you to use the data for free.
If you need to actually crawl and parse websites, for now you’ll probably find you have to write specific parsers for the various news websites to make sure you get the right text. However, once more websites start using HTML5, it will be easier to pull out the relevant text through the use of the article tag.
To do the actual crawling, this previous question can point you in some useful directions.