Question

I’ve written a script that uses two thread pools of ten threads each to pull in data from an API. The thread pool implements this code on ActiveState. Each thread pool is monitoring a Redis database via PubSub for new entries. When a new entry is published, python passes the data to a function that uses python’s Subprocess.POpen to execute a PHP shell to do the actual work of calling the API.

This system of launching PHP shells is necessary for functionality with my PHP web app, so launching PHP shells with Python can’t be avoided.

This script will only be running on Linux servers.

How do I control the niceness (scheduling priority) of the application’s threads?

Edit:

It seems controlling scheduling priority for individual threads in Python isn’t possible. Is there a python solution, or at the very least a UNIX command I can run along with my script, to control the priority?

Edit 2:

Well I didn’t end up finding a python way to handle it. I’m just running my script with nice now like this:

nice -n 19 python MyScript.py

Answer 1

I believe that threading priority is not controllable in python due to how they are implemented using a global interpreter lock (GIL). Having said that, even if you could give one thread more CPU processing priority, the python implementation that hands around the GIL would not be aware of this as it handed around the GIL. If you were able to increase niceness in a single thread in your pool (say it is doing a more important job) you would need to use your own implementation of locks to give the higher priority thread access to the GIL more often.

A google search returns this article which I believe is similar to what you are asking

Explains why it doesnt work
http://www.velocityreviews.com/forums/t329441-threading-priority.html

Explains the workaround I was suggesting
http://bytes.com/topic/python/answers/645966-setting-thread-priorities