Question

using the COPY command in postgresql 8.4 to export data of type bytea, this would work fine for a single record:

copy (SELECT encode(test_column, 'hex') FROM test_table LIMIT 1) TO '/home/user/file.hex'

after that, i use the xxd command to obtain the appropriate file type.
sorry in advance if this is a very noob question, but is it possible to export all the records into file-1,file-2…file-n? what’s the right syntax or script? i cant seem to find it in the man of postgresql copy.
for example, this is a bash script for a single row:

#!/bin/bash

psql \  
    -P t \  
    -P format=unaligned \  
    -X \  
    -U myuser \  
    -h myhost \  
    -c "select my_bytea_col from my_table where id=1" \  
    mydb \  
| xxd -r -p > dump.txt  

however, i need to execute it on all ids and have them saved under different files each such as file1.txt, file2.txt, file3.txt.. etc..
what could also be very interesting is the ability to use another column’s data (such as id) to name the resulting hex file accordingly.

Answer 1

I’d use a scripting language with direct PostgreSQL support for this.

This does the job quite nicely given a database named regress containing a table called files with columns fileid (an integer) and filedata (a bytea). It writes the data to files in the directory specified by the outdir variable, in files named according to a pattern like {filenameprefix}{fileid}{filenamesuffix}.

#!/usr/bin/env python3
import os
import sys
import psycopg2

outdir = "files"
filenameprefix = "f"
filenamesuffix = ""

def main():
        os.makedirs(outdir, exist_ok=True)
        conn = psycopg2.connect("dbname=regress")
        curs = conn.cursor();
        curs.execute("SELECT fileid, filedata FROM files;")
        for (fileid, filedata) in curs:
            fn = filenameprefix+str(fileid)+filenamesuffix
            f = open(os.path.join(outdir, fn), "wb")
            f.write(filedata)
            f.close()
        conn.close();

if __name__ == '__main__':
        main()

I used this code to create the test environment referred to in the above script:

#!/usr/bin/env python3
import psycopg2
conn = psycopg2.connect("dbname=regress")
curs = conn.cursor()
curs.execute('CREATE TABLE files(fileid serial primary key, filedata bytea);')
for i in range(1,10)
    curs.execute('INSERT INTO  files(filedata) VALUES (%s);', (psycopg2.Binary(open('/dev/urandom','rb').read(1024)),))

Edit: To make it the same as the naming you used in your question, which I just assumed was generic anonymized names, change:

• regress -> whatever your database name is
• files -> my_table
• fileid -> id
• filedata -> my_bytea_col

The equivalent to this script can be done in bash, it’s just cumbersome. You’d need to use a co-process to do it in a transaction-safe way. You’d need to create a psql co-process and open a SERIALIZABLE transaction in it, then SELECT fileid FROM files into a shell variable. Loop over the shell variable and for each ID, SELECT filedata FROM files into the xxd pipe to a filename created with string interpolation on the ID.

IMO: Not worth the hassle, use a scripting language that gives you direct PostgreSQL access.

If you don’t need it transaction safe it’s a little easier; just fetch the IDs in one psql invocation, then the file data in subsequent ones. It’ll be slower and won’t be transaction safe but it’ll be easier.