Question

I have an application written in Python. It collects and parses data from certain sources (RSS, Atom, Twitter, etc.) and if new data is found, it’s saved to a MySQL database.

I have many sources defined on the application itself, and not all can be checked on the same time basis because of restrictions (Twitter and its 350 queries per hour, for example), so the idea is to cron them separately by passing sources as arguments.

myscript.py –update –source ABC (Fetch, parse, check and save data from ABC source)
myscript.py –update –source XYZ (Fetch, parse, check and save data from XYZ source)

So far this point, I got that working fine on my local cron, but I’d like to get this running on Google App Engine, with something like this:

cron.yaml:

cron:
- description: update source ABC
  url: /myscript.py --source ABC
  schedule: every 5 minutes
  login: admin
- description: update source XYZ
  url: /myscript.py --source XYZ
  schedule: every 12 minutes
  login: admin

I haven’t found any way to get this done on App Engine Cron documentation. Any workarounds?

Answer 1

Put the source variable in your url path:

cron:
- description: update source ABC
  url: /cron/ABC/
  schedule: every 5 minutes
  login: admin
- description: update source XYZ
  url: /cron/XYZ/
  schedule: every 12 minutes
  login: admin

webapp.WSGIApplication([('/cron/([^/]+)/', CronHandler)])    

class CronHandler(webapp.RequestHandler):
    def post(self, source): 
            #do something with source variable...