I have written a Django app that makes use of Python threading to create a web spider, the spider operates as a series of threads to check links.
When I run this app using the django test server (built in), the app runs fine and the threads seem to start and stop on time.
However, running the app on Apache it seems the threads aren’t kicking off and running (after about 80 seconds there should be a queued database update and these changes aren’t occuring).
Does anyone have an idea what I’m missing here?
— Edit: My question is, how does Apache handle threaded applications, i.e. is there a limit on how many threads can be run from a single app?
Any help would be appreciated!
Most likely, you are missing the creation of new processes. Apache will not run in a single process, but fork new processes for requests every now and then (depending on a dozen or so configuration parameters). If you run django in each process, they will share no memory, and the results produced in one worker won’t be visible to any of the others. In addition, the Apache process might terminate (on idle, or after a certain time), discarding your in-memory results.