Question

From what I’ve read around the interwebs and here on SO, os.walk is one of the best bets for finding all subdirs and files in a directory, but the question I have is, if I only want to recursively find all the subdirectories, is os.walk the fastest solution?

I suppose in addition my question is since the call to os.walk involves for root, dirs, files in os.walk(path), does os.walk actually look at all the files if you don’t necessarily call for it?

Presently, my code is:

for root, dirs, files in os.walk(path):
    for x in dirs:
        DoStuffHere

But it sure seems slow on a folder with many many subdirs and files.

Thanks.

Answer 1

I’m not 100% for certain on all aspects of this but from my understanding:

In general the file listing is already included in the directory metadata so when you look up a directory the data is already there (Linux, not sure bout windows). So this means that os.walk is probably the fastest/simplest way of doing this.

Also again without profiling do you know if os.walk is really where the slow down is? Remember the general advice is to code up your application/project and then if its too slow, start profiling to find the slow part and re-factor them, etc…

On os.walk I’m able to os.walk roughly over a few thousand directory + 70 thousand files within a few second so its going to probably be plenty fast for your need.