Question

I build a django based geolocation service which determines user’s location by their IP address. The first thing I need to do is to insert ip data into my database.

I used following code(simplified) to insert records into my db:

 for ipLoc in ipSeeker.ipLocationList:
     placeName =ipLoc.country + ipLoc.area
     IPLog.objects.create(
                startIP = int_to_dqn(ipLoc.startIP), 
                endIP = int_to_dqn(ipLoc.endIP), 
                place = placeName
     ).save()

The ipLocationList has approx 400k ip records. And my script only insert 20k records in 20 minutes. It is tooooooo slow and could not be accepted.

So my question is: where is the bottleneck and how could I make it faster?

Thanks in advance!

Answer 1

Use raw sql and transactions for django-side: https://docs.djangoproject.com/en/dev/topics/db/sql/ along with the Copy command from Postgres!

Whats the fastest way to do a bulk insert into Postgres?