Question

I want to develop an open source library, for a fast efficient file storage (under one large file, and index file) like NFileStorage. why i want to do this ?
A. under my line of work something like that waS needed.
B. our DBA said its not efficient to store files under the DB.
C. Its a good practice for me.

I am looking for a good article for file indexes
can you recommend one ?

what is your general idea ?

Answer 1

It may not be efficient to store files inside a database, however databases like SQL Server have the concept of FileStreams where it actually stores it on the local file system instead of placing it in the database file itself.

In my opinion this is a bad idea for a project.

You are going to run into exactly the same problem that databases have with storing all of the uploaded files inside the same single file… which is why some of them have moved away from this for binary / large objects and instead support alternative methods.

Some of the problems you will have to deal with include:

1. Allocating additional disk space for your backing file to store newly uploaded documents.
2. Permanently removing “files” from your storage and resizing / compressing the backing file.
3. Multi-user access / locks.
4. Failure recovery. Such as when you encounter a bad block on the drive and it hoses your backing file.
5. Transactional support.

Items 1 and 2 cause an increase in the amount of time it takes to write a “file” to your data store. Items 3, 4 and 5 are already supported by network file systems so you’re just recreating the wheel.

In short you’re going to have to either write your own file system or write your own DBMS. Neither of which I would consider “good practice” for 99% of real world applications. It might be worthwhile if your goal is to work for Seagate.. But even then they’d probably look at you funny.

If you are truly interested in the most efficient method of file storage, it is quite simply to purchase a SAN array and push your files to it while keeping a pointer to the file/location in your database. Easy to back up, fast to store files, much cheaper than spending developer time trying to figure out how to write your own file system and certainly 100% supported and understandable by future devs.