Question

I am trying to upload a moderetly sized csv file to google app engine using the bulkloader functionality, and it appears to die some of the way through with the following result:

[INFO    ] Logging to bulkloader-log-20110328.181531
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20110328.181531.sql3
[INFO    ] Connecting to notmyrealappname.appspot.com/_ah/remote_api
[INFO    ] Starting import; maximum 10 entities per post
...............................................................[INFO    ] Unexpected thread death: WorkerThread-7
[INFO    ] An error occurred. Shutting down...
.........[ERROR   ] Error in WorkerThread-7: <urlopen error [Errno -2] Name or service not known>

[INFO    ] 1740 entites total, 0 previously transferred
[INFO    ] 720 entities (472133 bytes) transferred in 32.3 seconds
[INFO    ] Some entities not successfully transferred

It uploads about 700 of the 19k entries I am trying to upload, and I am wondering why it fails. I checked the csv file for errors like additional commas that could throw off the python csv reader and non ascii characters have been stripped out.

Answer 1

Lifting the batch limit (batch_size) and rps limit (rps_limit) works, I use 1000 for the batch size and an rps limit of 500:

appcfg.py upload_data --url= --application= --filename=  --email= --batch_size=1000 --rps_limit=500