Question

I am using python scripts to load data to a database bulk loader.

The input to the loader is stdin. I have been unable to get the correct syntax to call the unix based bulk loader passing the contents of a python list to be loaded.

I have been reading about Popen and PIPE but they have not been behaving as i expect.

The python list contains database records to be bulkloaded. In linux it would look similar to this:

echo "this is the string being written to the DB" | sql -c "COPY table FROM stdin"

What would be the correct way replace the echo statement with a python list to be used with this command ?

I do not have sample code for this process as i have been experimenting with the features of Popen and PIPE with some very simple syntax and not obtaining the desired result.

Any help would be very much appreciated.

Thanks

Answer 1

If your data is short and simple, you could preformat the entire list and do it simple with subprocess like this:

import subprocess
data = ["list", "of", "stuff"]
proc = subprocess.Popen(["sql", "-c", "COPY table FROM stdin"], stdin=subprocess.PIPE)
proc.communicate("\n".join(data))

If the data is too big to preformat like this, then you can attempt to use the stdin pipe directly, though subprocess module is flaky when using the pipes if you need to read from stdout/stderr too.

for line in data:
    print >>proc.stdin, line