Question

I’m trying to write a script which lists a directory and creates an SQL script to insert these directories, problem is I only want to insert new directories, here is what I have so far:

#If file doesn't exist add the search path test
if [ ! -e  /home/aydin/movies.sql ] 
then
    echo "SET SEARCH_PATH TO noti_test;" >> /home/aydin/movies.sql;
fi
cd /media/htpc/
for i in *
do
    #for each directory escape any single quotes
    movie=$(echo $i | sed "s:':\\\':g" )
    #build sql insert string
    insertString="INSERT INTO movies (movie) VALUES (E'$movie');";
    #if sql string exists in file already   
    if grep -Fxq "$insertString" /home/aydin/movies.sql
    then
        #comment out string
        sed -i "s/$insertString/--$insertString/g" /home/aydin/movies.sql
    else
        #add sql string
            echo $insertString >> /home/aydin/movies.sql;
    fi
done;
#execute script
psql -U "aydin.hassan" -d "aydin_1.0" -f /home/aydin/movies.sql;

It seems to work apart from one thing, the script doesn’t recognise entries with single quotes in them, so upon running the script again with no new dirs, this is what the file looks like:

--INSERT INTO movies (movie) VALUES (E'007, Moonraker (1979)');
--INSERT INTO movies (movie) VALUES (E'007, Octopussy (1983)');
INSERT INTO movies (movie) VALUES (E'007, On Her Majesty\'s Secret Service (1969)');  

I’m open to suggestions on a better way to do this also, my process seems pretty elongated and inefficient 🙂

Answer 1

Script looks generally good to me. Consider the revised version (untested):

#! /bin/bash
#If file doesn't exist add the search path test
if [ ! -e  /home/aydin/movies.sql ] 
then
    echo 'SET search_path=noti_test;' > /home/aydin/movies.sql;
fi
cd /media/htpc/
for i in *
do
    #build sql insert string - single quotes work fine inside dollar-quoting
    insertString="INSERT INTO movies (movie) SELECT \$x\$$movie\$x\$
WHERE NOT EXISTS (SELECT 1 FROM movies WHERE movie = \$x\$$movie\$x\$);"

    #no need for grep. SQL is self-contained.
    echo $insertString >> /home/aydin/movies.sql
done

#execute script
psql -U "aydin.hassan" -d "aydin_1.0" -f /home/aydin/movies.sql;

• To start a new file, use > instead of >>

• Use single quotes ' for string constants without variables to expand

• Use PostgreSQL dollar-quoting so you don’t have to worry about single-quotes in the strings. You’ll have to escape the $ character in the shell to remove its special meaning in the shell.
  Use an “impossible” string for the dollar-quote, so it cannot appear in the string. If you don’t have one, you can test for the quote-string and alter it in the unlikely case it should be matched, to be absolutely sure.

• Use SELECT .. WHERE NOT EXISTS for the INSERT to automatically prevent already existing entries to be re-inserted. This prevents duplicate entries in the table completely – not just among the new entries.

• An index on movies.movie (possibly, but not necessarily UNIQUE) would speed up the INSERTs.