Home/ Questions/Q 6628643
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-25T22:08:55+00:00

Background: While coding on GAE’s local Development Web Server, user need to upload Mega-level

  • 0

Background:

While coding on GAE’s local Development Web Server, user need to upload Mega-level datas and store (not straight forward store, but need many format check and translate) them into Datastore using deferred library.

Usually about 50,000 entities, CSV File size is about 5MB, and I tried to insert 200 entities each time using deferred library.

And I used python.

Problem

The development server is really slow that I need to wait one/more hours to finish this upload process.

I used –use_sqlite option to speed up the development web server.

Question:

Is there any other method or tuning that can make it faster?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    appengine-mapreduce is definitely an option for loading CSV files. Use blobstore to upload CSV file and then setup BlobstoreLineInputReader mapper type to load data into datastore.

    Some more links: Python Guide to mapreduce reader types is here. The one of interest is BlobstoreLineInputReader. The only input it requires is the key to the blobstore record containing uploaded CSV file.

    • 0