Question

I’m writing an aggregation application which scrapes data from a couple of web sources and displays that data with a novel interface. The sites from which I’m scraping update every couple of minutes, and I want to make sure the data on my aggregator is up-to-date.

What’s the best way to periodically submit fresh data to my App Engine application from an automated script?

Constraints:

1. The application is written in Python.

2. The scraping process for each site takes longer than one second, thus I cannot process the data in an App Engine handler.

3. The host on which the updater script would run is shared, so I’d rather not store my password on disk.

4. I’d like to check the code for the application into to our codebase. While my associates aren’t malicious, they’re pranksters, and I’d like to prevent them from inserting fake data into my app.

5. I’m aware that App Engine supports some remote_api thingey, but I’d have to put that entry point behind authentication (see constraint 3) or hide the URL (see constraint 4).

Suggestions?

Answer 1

Write a Task Queue task or an App Engine cron job to handle this. I’m not sure where you heard that there’s a limit of 1 second on any sort of App Engine operations – requests are limited to 30 seconds, and URL fetches have a maximum deadline of 10 seconds.