Question

I have a script that is getting the GeoIP locations of various ips, this is run daily and I’m going to expect to have around ~50,000 ips to look up.

I have a GeoIP system set up – I just would like to eliminate having to run wget 50,000 times per report.

What I was thinking is, there must be some way to have wget open a connection with the url – then pass the ips, that way it doesn’t have to re-establish the connection.

Any help will be much appreciated.

Answer 1

If you give wget several addresses at once, with consecutive addresses belonging to the same HTTP/1.1 (Connection: keep-alive) supporting server, wget will re-use the already-established connection.

If there are too many addresses to list on the command line, you can write them to a file and use the -i/--input-file= option (and, per UNIX tradition, -i-/--input-file=- reads standard input).

There is, however, no way to preserve a connection across different wget invocations.