There is a website, which provides the data I want to get. However download form
allows to download it in small portions, so I need to fill it and press “download”
button about 18000 times. On a forum of this web site it was said by administrator,
that downloading the data using this form is the only way to get it and some people write
scripts to automate the process, such scripts are eligible if they do not start multiple
downloads simultaneously. I want to write such script, but I don’t know, where to start, because I’m not familiar with interaction of a web page with a server.
I have some experience writing C/C++ programs and I know a bit of bash scripting. I know basics of html, but I don’t really understand, how html forms work. I usually work in Ubuntu, if it is needed I have Windows 7 installed too.
Please, tell me, where to start: which tools can I use and which notions should I learn to solve this task, which books/tutorials/articles may be helpful for my education in that direction.
Particularly, are languages I know (C++, bash script) suitable for this scripting or otherwise is it better to learn some other languages? Which programs do you use to understand, what should a script send to a server and how should it receive a file back?
This kind of thing is called web scraping. There are multiple ways of doing it. A command line tool called curl allows you to fetch resources from a website in the same way as your browser would. It is however scriptable and you can pass parameters to it to make it work like a browser.
Ordinarily, I’d do this using a scripting language like Python but it is possible, although harder, to do it using a bash script and curl. You’ll need to figure out what exactly you’re submitting to the website when you click on the submit button. The firefox plugin firebug can help you figure this out and then do the same thing using curl. Put the curl invocation in a loop and vary the parameters to simulate the 18000 clicks.