Home/ Questions/Q 7028021
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-28T00:19:51+00:00

It’s well known that in Windows a directory with too many files will have

  • 0

It’s well known that in Windows a directory with too many files will have a terrible performance when you try to open one of them. I have a program that is to execute only in Linux (currently it’s on Debian-Lenny, but I don’t want to be specific about this distro) and writes many files to the same directory (which acts somewhat as a repository). By “many” I mean tens each day, meaning that after one year I expect to have something like 5000-10000 files. They are meant to be kept (once a file is created, it’s never deleted) and it is assumed that the hard disk has the required capacity (if not, it should be upgraded). Those files have a wide range of sizes, from a few KB to tens of MB (but not much more than that). The names are always numeric values, incrementally generated.
I’m worried about long-term performance degradation, so I’d ask:

  • Is it OK to write all to the same directory? Or should I think about creating a set of subdirectories for every X files?
  • Should I require a specific filesystem to be used for such directory?
  • What would be the more robust alternative? Specialized filesystem? Which?
  • Any other considerations/recomendations?
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It depends very much on the file system.

    ext2 and ext3 have a hard limit of 32,000 files per directory. This is somewhat more than you are asking about, but close enough that I would not risk it. Also, ext2 and ext3 will perform a linear scan every time you access a file by name in the directory.

    ext4 supposedly fixes these problems, but I cannot vouch for it personally.

    XFS was designed for this sort of thing from the beginning and will work well even if you put millions of files in the directory.

    So if you really need a huge number of files, I would use XFS or maybe ext4.

    Note that no file system will make “ls” run fast if you have an enormous number of files (unless you use “ls -f”), since “ls” will read the entire directory and the sort the names. A few tens of thousands is probably not a big deal, but a good design should scale beyond what you think you need at first glance…

    For the application you describe, I would probably create a hierarchy instead, since it is hardly any additional coding or mental effort for someone looking at it. Specifically, you can name your first file “00/00/01” instead of “000001”.

    • 0