I made a dynamic site that has over 20,000 pages and once a page is created there is no need to update it for at-least one month or even a year. So I’m caching every page when it is first created and then delivering it from a static html page
I’m running a php script (whole CMS is on PHP) if (file_exists($filename)) to first search for the filename from the url in cache-files directory and if it matches then deliver it otherwise generate the page and cache it for latter use. Though it is dynamic but still my url does not contain ?&=, I’m doing this by - and breaking it into array.
What I want to know is will it create any problem to search for a file from that huge directory?
I saw a few Q/A like this where it says that there should not be problem with number of files I can store on directory with ext2 or ext3 (I guess my server has ext3) file system but the speed of creating a new file will decrease rapidly after there are files over 20-30,000.
Currently I’m on a shared host and I must cache files. My host a soft limit of 100,000 files in my whole box which is good enough so far.
Can someone please give me any better idea about how to cache the site.
You shouldn’t place all of the 20K files in a single directory.
Divide them into directories (by letter, for example), so you access:
etc.
That would allow you to place more files with less constraints on the file-system, which would increase the speed. (since the FS doesn’t need to figure out where your file is in the directory along with other 20k files, it needs to look in about a hundred)