Question

Im transfering txt files into a sqlite3 database using sqlite-ruby, each txt file is about 3,5 MB which ruby reads really fast, but populating the database takes more than 10 minutes per file. I don´t know if this is normal, i guess not. Can somebody tell me a faster way to do this or if im doing something crazy? code and txt scheme below, thanks.

EDIT: more than 30 min per file…

require 'sqlite3'

db = SQLite3::Database.new( "firstTest.db" )

filename="file1.txt" # this file is 3.5 MB

streetArray=[]
mothertablename="tmother"
coordstablename="tcoords"

db.execute("create table '#{mothertablename}' (id INTEGER,stName TEXT,stBC TEXT,stPrice INTEGER,stUrl TEXT);")
db.execute("create table '#{coordstablename}' (id INTEGER,num INTEGER,xcoord DOUBLE,ycoord DOUBLE);")

f=[]
File.open("_history/" + "#{filename}").each{|line| f.push(line)}

counter=0

for i in 1...f.length
    f[i].gsub!("'","") #get rid of comma character
    ptsArray=f[i].split(';')

    if ptsArray[0]==ptsArray[1]
        streetArray.push(f[i])
    end

    if ptsArray[0]=="next\n"
        counter +=1
        namestreetArray=f[i-1].split(';')
        stname=nil
        stprice=nil
        stbc=nil
        sturl=nil
        if namestreetArray[0]=="name"
            stname=namestreetArray[1]
            if namestreetArray.length>2
                stprice=namestreetArray.last
            else
                stprice=nil
            end
        end

        db.execute(  "insert into '#{mothertablename}' (id, stName, stBC, stPrice, stUrl) 
        values ('#{counter}','#{stname}','#{stbc}','#{stprice}','#{sturl}');")

        localcounter=0

        streetArray.each do |coord|
            localcounter +=1
            xcoord=coord.split(";")[3].to_f
            ycoord=coord.split(";")[2].to_f
            db.execute(  "insert into '#{coordstablename}' (id, num, xcoord, ycoord) 
            values ('#{counter}','#{localcounter}','#{xcoord}','#{ycoord}');")
        end
        streetArray.clear
    end
end

here´s the txt scheme

206358589;206358589;37.3907322;-5.9885633
195966401;195966401;37.3909862;-5.988974
195969491;195969491;37.3910908;-5.9891081
195969493;195969493;37.3911863;-5.9893141
195969494;195969494;37.3912954;-5.9895115
813726831;813726831;37.3914352;-5.98973
name;Calle Descalzos;3085
next
440230342;440230342;37.3918677;-5.9905477
813726823;813726823;37.3916168;-5.9905285
192037929;192037929;37.3916184;-5.9905125
195970140;195970140;37.391872;-5.990398
440230342;440230342;37.3918677;-5.9905477
next
192009475;192009475;37.3875271;-5.9937949
710633000;710633000;37.3875013;-5.9941761
name;Calle Felipe Perez;
next
195982576;195982576;37.387349;-5.9937755
308836571;308836571;37.3873649;-5.9936472
next
...   

Answer 1

See the faq of sqlite “INSERT is really slow – I can only do few dozen INSERTs per second”

In short, you can speed up the sqlite by two ways:

• Group multiple INSERT statements with BEGIN…COMMIT to make the inserts into one transaction. In ruby, you should do something like:

  database.transaction do |db| db.execute( "insert into table values ( 'a', 'b', 'c' )" ) ... end 
  

• Or simply run PRAGMA synchronous=OFF.

And remember to prepare the statement before executing it.