I’m writing a document editing web service, in which documents can be edited via a website, or locally and pushed via git. I’m trying to decide if the documents should be stored as individual documents on the filesystem, or in a database. The points I’m wondering are:
- If they’re in a database, is there any way for git to see the documents?
- How much higher are the overheads using the filesystem? I assume the OS is doing a lot more work. How can I alleviate some of this? For example, the web editor autosaves, what would the best way to cache the save data be, to minimise writes?
- Does one scale significantly better or worse than the other? If all goes according to plan, this will be a service with many thousands of documents being accessed and edited.
If the documents go into a database, git can’t directly see the documents. git will see the backing storage file(s) for the database, but have no way of correlating changes there to changes to files.
The overhead of using the database is higher than using a filesystem, as answered by Carlos. Databases are optimized for transactions, which they’ll do in memory, but they have to hit the file. Unless you program the application to do database transactions at a sub-document level (Eg: changing only modified lines), the database will give you no performance improvement. Most modern filesystems do caching and you can ‘write’ in a way that will sit in RAM rather than going to your backing stoage as well. You’ll need to manage the granularity of the ‘autosaves’ in your application (every change? every 30 seconds? 5 minutes?), but really, doing it at the same granularity with a database will cause the same amount of traffic to the backing store.
I think you intended to ask “does the filesystem scale as well as the database”? 🙂 If you have some way to organize your files per-user, and you figure out the security issue of a particular user only being able to access/modify the files they should be able to (which are doable imo), the filesystem should be doable.