Question

Using Python (2.7)and sqlite (3) I am trying to copy results of a query into a table.
Because the result of the query is very large, I would like to use “fetchmany” in batches.
The query works fine, retrieving the results in the batches as well.
The problem is that when I try to copy the results in the table, it stops after the first batch.

I suspect that the problem is the place of the cursor.

How does one returns the cursor in python ?

P.S: I have seen here many posting about cursor (closing) but haven’t seen the answer to my question. Please note also that I am new to Python, so apologies if the question is trivial.

Here pieces of my codes: (example)

                import sqlite3

                dbLocation = 'P:/XXX/db1.db'
                connection = sqlite3.connect(dbLocation)
                cursor = connection.cursor()

                strSQLStatement = """
                        SELECT
                            whatever1,
                            whaterver2
                        from wherever
                        LIMIT 10"""

                cursor.execute(strSQLStatement)


                #the following codes works 
                # printing the 10 results


                while True:
                    results = cursor.fetchmany(2)
                    if not results:
                      break
                    print results   


                #the following codes does NOT work  
                # Only 2 results are processed 

                while True:
                    results = cursor.fetchmany(2)
                    if not results:
                      break
                    print results   
                    cursor.executemany ('INSERT INTO NewTable (?,?)',results)
                    connection.commit()

Answer 1

Your call to executemany() on the original cursor clobbers what was there before. Create a second cursor to perform the insert.