See EDIT#2 below for likely root cause
My Django threads/processes (same thing happens when using both ‘method=threaded’ and ‘method=prefork’), every 10 mins or so, randomly build up like so:
https://i.stack.imgur.com/ihuNR.png e.g lots of these processes:
python ./manage.py runfcgi method=threaded host=localhost port=8888
Then in a minute or so, the threads/processes build up enough to take out my site.
The site load is pretty minimal, but the traffic is fairly high, so I believe it’s some sort of blocking issue.
Any ideas on how to solve, or even effectively debug this issue?
Thanks.
EDIT: I also use Redis a lot from many of my Django views. I’m now beginning to look into the possibility of a Redis deadlock issue being the root of the problem…
EDIT#2: I’m seeing that Redis gets pinned at 100% CPU when the issue starts, which makes me believe a Redis deadlock of sorts is the core issue.
I also got this error:
“(error) LOADING Redis is loading the dataset in memory”
when trying to fully restart Redis via this commmand:
./src/redis-cli flushall
IMPORTANT: I’m using Redis 2.2.7 on a separate EC2 micro instance than my Django web application.
Turned out to be an obscure bug in Redis.
A Redis update, Redis 2.2.7 -> Redis 2.4.16 fixed the problem.