Question

I want to post a http document that is password protected.
It has a username and password login page before you can access the actual document.
I tried doing this

curl 
 -u username:password 
 "http://localhost:8983/solr/update/extract?literal.id=doc900&commit=true" 
 -F stream.url=http://somewebsite.com/docs/DOC2609

but it just indexes the login page only.

Answer 1

My guess is that the authentication method this page of yours uses is not one of the HTTP authentication methods that curl supports out-of-the-box.

The HTTP authentication is basic — some might say trivial — and it is fundamentally insecure in its most basic form (literally, ‘basic’ mode HTTP authentication sends the users’ password over the wire in plain text, as one of the request headers). I only ever see it used in conjunction with HTTPS, and even that’s pretty rare these days.

HTTP authentication is generally eschewed in favor of either an HTTP PUT or POST request over HTTPS, usually fortified with some kind of cross-site request forgery (CSRF) prevention measure, or an OAUTH2 strategy (which typically defers the issue to the authentication broker’s implementation).

Your problem is, curl doesn’t know about any of this. If you program in python, you may find the requests package helpful (and more modern than curl in its design to boot). Solr won’t care either way, if you are calling its HTTP API properly.