Im trying to grab a HTML source code from a webpage which needs user to login with username and password, the code are all done in the back end, i can not use “?username=xx&pw=xxx” to get the html code.
There are many apps that does this kind of things, such as getting usage from a phone provider or view bills.
I would like to know how do people implement this. do they create some kind of robot that input those username into the textfield and click submit? and is there a way to download the html straight from the server, rather than create our own server to retrieve it.
How do i set the username textfield on the webpage and click log in button to retrieve the next page?
Thanks in advance
Why not? If a user can submit their login creds over HTTP, then you can do the same. You’ll probably need to use POST instead of GET to submit those fields, and there may be other fields on the form that you’ll need to match. But the web server doesn’t (in principle) have any idea what sort of client is submitting login data to it.
Now, they might screen out older, non-supported browsers, in which case your USER_AGENT field will need to be fudged to some legal value. There may be some session timeout token in the form that you need to get first and provide in your submitted data. Who knows what else they need back from you to validate the login. But if a web browser can do it, you can do it from the HTTP client methods in your app.