Question

I wish to understand the way kernel works when a user/app tries to create a file in a directorty.

The background – We have a java applicaiton which consumes messages over JMS, processes it and then writes the XML to an outbound queue+a local directory. Yesterday we obeserved unsual delays in writing to the directory. On ‘ls|wc -l’ we found >300,000 files in there. Did a quick strace on the process and found it full of mutex calls (More than 3/4 calls in the strace were mutex).

So i thought that new file creation is taking time becasue the system has to every time check certain things (e.g name of files to make sure that the new file with a specific name can be created) amongst 300,000 files and then create a file.

I cleared the directory and the applicaiton resumed to normal service levels.

My questions

1. Was my analysis correct (It seems cuz the app started working fine after a clear down)?
2. More imporatant, how does the kernel work when you try to creat a new file in directory.
3. Can the abnormal number of mutex calls be attributed to the high number of files in the directory?

Many thanks
J

Answer 1

Please read about the Linux Filesystem, i-nodes and d-nodes.

http://en.wikipedia.org/wiki/Inode_pointer_structure

The file system is organized into fixed-sized blocks. If your directory is relatively small, it fits in the direct blocks and things are fast. If your directory is not too big, it fits in the direct blocks and some indirect blocks, and is still reasonably fast. If your directory becomes too big, it spills into double indirect blocks and becomes slow.

Actual sizes depend on file system and kernel configuration.

Rule of thumb is to keep the directory under 12 blocks, depending on your block size. Many systems use 8K blocks; a fast directory is under 98,304 bytes.

A file entry is something like 16*4 bytes in size (IIRC), so plan on no more than 1500 files per directory as a practical upper limit.