Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Do you know any large dataset to experiment with Hadoop which is free/low cost?
Any pointers/links related are appreciated.
Preference:
At least one GB of data.
Production log data of webserver.
Few of them which I found so far:
Also can we run our own crawler to gather data from sites e.g. Wikipedia? Any pointers on how to do this is appreciated as well.
Few points about your question regarding crawling and wikipedia.
You have linked to the wikipedia data dumps and you can use the Cloud9 project from UMD to work with this data in Hadoop.
They have a page on this: Working with Wikipedia
Another datasource to add to the list is:
Using a crawler to generate data should be posted in a separate question to one about Hadoop/MapReduce I would say.