I am working on rewriting a Java web application to Rails which relies heavily on collections (100’s or 1000’s) of large (50-100MB) TIFF files. In the Java version, the user specifies a local root path (such as a mounted SAN drive) for these files in the application configuration, and they are read by the application using these paths. The application also writes new files to those paths.
Essentially, users must be able to add files to the application in two ways:
1) Specify a storage location as the ‘root’ for a collection of TIFFs, which could already contain many TIFFs. These are then processed.
2) Upload new files to an existing collection, which would then be written to the above path and processed.
I guess the gist of my question is: What is the standard way to store, retrieve, and write to such large files in the context of web applications? Should the availability of a local file system with enough storage space be assumed, or is there a better way to do it?
I would look into storing the files with paperclip or carrierwave. They are two great file upload and management gems that allow you to store your files in many different ways.
I have included links to two great sceencasts above and here are the github pages for paperclip and carrierwave.