Question

After a customer uploads an image I want to re-size an image to 8 different sizes and then send them up to S3 and save them to the file system. Obviously this could take a long time (seconds) and I would want to do it in a “multithreaded” way (I quote that because I don’t care if it’s actually multithreaded or if it’s some other functionality). My understanding is that because of the GIL in Python you cannot successfully use multithreading for things like this because it uses the same process and therefore works exactly the same as if you did it in your code’s normal flow (vs multithreading). Is there some other, better way of opening a new process in Python for this sort of task?

Answer 1

If you want to call an external process, use the subprocess module.

If on the other hand you want to side-step the GIL and spin of some Python task, use the multiprocessing module. It provides an interface very much like the threading package’s but utilizes subprocesses so you are not bound by the constraints of the GIL.