Question

I’m trying to insert strings read from a file into an sqlite database in Python. The strings have whitespace (newline, tab characters, and spaces) and also have appearances of single or double quotes. Here’s how I try to do it:

import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()

# Create table
c.execute('''CREATE TABLE test
             (a text, b text)''')

f = open("foo", "w")
f.write("hello\n\'world\'\n")
f.close()

testfield = open("foo").read()

# Insert a row of data
c.execute("INSERT INTO test VALUES ('%s', 'bar')" %(testfield))

# Save (commit) the changes
conn.commit()

I find that this fails with the error:

    c.execute("INSERT INTO test VALUES ('%s', 'bar')" %(testfield))
sqlite3.OperationalError: near "world": syntax error

How can I achieve this? Do the strings need to be escaped before insertion in the db, and if so how? thanks.

Answer 1

You use SQL parameters instead of string formatting:

c.execute("INSERT INTO test VALUES (?, 'bar')", (testfield,))

When using SQL parameters you let the database library handle the quoting, and even better, give the database to optimize the query and reuse the optimized query plan for multiple executions of the same basic query (with different parameters).

Last but not least, you are much better defended against SQL injection attacks as the database library knows best how to escape dangerous SQL-like values.

To quote the sqlite3 documentation:

Usually your SQL operations will need to use values from Python variables. You shouldn’t assemble your query using Python’s string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack (see http://xkcd.com/327/ for humorous example of what can go wrong).

Instead, use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.