Home/ Questions/Q 8465533
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T15:04:40+00:00

I’ve read that certain Python functions implemented in C, which I assume includes file.read(),

  • 0

I’ve read that certain Python functions implemented in C, which I assume includes file.read(), can release the GIL while they’re working and then get it back on completion and by doing so make use of multiple cores if they’re available.

I’m using multiprocess to parallelize some code and currently I’ve got three processes, the parent, one child that reads data from a file, and one child that generates a checksum from the data passed to it by the first child process.

Now if I’m understanding this right, it seems that creating a new process to read the file as I’m currently doing is uneccessary and I should just call it in the main process. The question is am I understanding this right and will I get better performance with the read kept in the main process or in a separate one?

So given my function to read and pipe the data to be processed:

def read(file_path, pipe_out):
    with open(file_path, 'rb') as file_:
        while True:
            block = file_.read(block_size)
            if not block:
                break
            pipe_out.send(block)
    pipe_out.close()

I reckon that this will definitely make use of multiple cores, but also introduces some overhead:

multiprocess.Process(target=read, args).start()

But now I’m wondering if just doing this will also use multiple cores, minus the overhead:

read(*args)

Any insights anybody has as to which one would be faster and for what reason would be much appreciated!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Okay, as came out by the comments, the actual question is:

    Does (C)Python create threads on its own, and if so, how can I make use of that?

    Short answer: No.

    But, the reason why these C-Functions are nevertheless interesting for Python programmers is the following. By default, no two snippets of python code running in the same interpreter can execute in parallel, this is due to the evil called the Global Interpreter Lock, aka the GIL. The GIL is held whenever the interpreter is executing Python code, which implies the above statement, that no two pieces of python code can run in parallel in the same interpreter.

    Nevertheless, you can still make use of multithreading in python, namely when you’re doing a lot of I/O or make a lot of use of external libraries like numpy, scipy, lxml and so on, which all know about the issue and release the GIL whenever they can (i.e. whenever they do not need to interact with the python interpreter).

    I hope that cleared up the issue a bit.

    • 0