Question

I’m tasked with writing an application that acts as a central storage point for files (usually document formats) as provided by other applications. It also needs to take commands like “file 395 needs a copy in X format”, at which point some work is offloaded to a 3rd party application. I’m having trouble coming up with a strategy for this.

I’d like to keep the design as simple as possible, so I’d like to avoid big extra frameworks or techniques like threads for as long as it makes sense.

The clients are expected to be web applications (for example, one is a django application that receives files from our customers; the others are not yet implemented). The platform it will be running on is likely going to be Python on Linux, unless I have a strong argument to use something else.

In the beginning I thought I could fit the information I wanted to communicate in the filenames, and let my application parse the filename to figure out what it needed to do, but this is proving too inflexible with the amount of information I’m realizing I need to make available.

Another idea is to pair FTP with a database used as a communication medium (client uploads a file and updates the database with a command as a row in a table) but I don’t like this idea because adding commands (a known change) looks like it will require adding code as well as changing database schemas. It will also muddy up the interface my clients will have to use.

I looked into Pyro to let applications communicate more directly but I don’t like the idea of running an extra nameserver for this one purpose. I also don’t see a good way to do file transfer within this framework.

What I’m looking for is techniques and/or technologies applicable to my problem. At the simplest level, I need the ability to accept files and messages with them.

Answer 1

You should be able to implement this in a fairly RESTful way with HTTP PUT and GET operations. This would be very useful for several reasons:

• Able to link to the storage directly from internal websites
• Easy testing
• Lots of libraries available to help you implement this
• No need to worry about what platform the client application uses

I would suggest implementing it so that getting a file in a specific format is as simple as navigating to:

http://www.myserver.com/filestore/documents/docname&format=xxx

Within the server of course I would use a database of documents, file formats and cached versions of already converted files. I would invoke the third party translators on demand, i.e. when a request comes in for a document in a specific format and the document is not already in the cache.