Question

i am inserting records to sql server from python using pymssql. The database takes 2 milliseconds to execute a query, yet it insert 6 rows per second. The only problem is at code side. how to optimize following code or what is the fastest method to insert records.

def save(self):
    conn = pymssql.connect(host=dbHost, user=dbUser, 
                           password=dbPassword, database=dbName, as_dict=True)
    cur = conn.cursor()

    self.pageURL = self.pageURL.replace("'","''")

    query = "my query is there"               
    cur.execute(query)

    conn.commit() 
    conn.close()

Answer 1

It looks like you’re creating a new connection per insert there. That’s probably the major reason for the slowdown: building new connections is typically quite slow. Create the connection outside the method and you should see a large improvement. You can also create a cursor outside function and re-use it, which will be another speedup.

Depending on your situation, you may also want to use the same transaction for more than a single insertion. This changes the behaviour a little — since a transaction is supposed to be atomic and either completely succeeds or completely fails — but committing a transaction is typically a slow operation, because it has to be certain the whole operation succeeded.