Question

I am designing a web-based file-managment system that can be conceptualised as 3 different servers:

1. The server that hosts the system interface (built in PHP) where users ‘upload’ and manage files (no actual files are stored here, it’s all meta).
2. A separate staging server where files are placed to be worked on.
3. A file-store where the files are stored when they are not being worked on.

All 3 servers will be *nix-based on the same internal network. Users, based in Windows, will use a web interface to create an initial entry for a file on Server 1. This file will be ‘uploaded’ to Server 3 either from the user’s local drive (if the file doesn’t currently exist anywhere on the network) or another network drive on the internal network.

My question relates to the best programmatic approach to achieve what I want to do, namely:

When a user uploads a file (selecting the source via a web form) from the network, the file is transferred to Server 3 as an inter-network transfer, rather than passing through the user (which I believe is what would happen if it was sent as a standard HTTP form upload). I know I could set up FTP servers on each machine and attempt to FXP files between locations, but is this preferable to PHP executing a command on Server 1 (which will have global network access), to perform a cross-network transfer that way?

The second problem is that these are very large files we’re talking about, at least a gigabyte or two each, and so transfers will not be instant. I need some method of polling the status of the transfer, and returning this to the web interface so that the user knows what is going on.

Alternatively this upload could be left to run asyncrhonously to the user’s current view, but I would still need a method to check the status of the transfer to ensure it completes.

So, if using an FXP solution, how could polling be achieved? If using a file move/copy command from the shell, is any form of polling possible? PHP/JQuery solutions would be very acceptable.

My final part to this question relates to windows network drive mapping. A user may map a drive (and select a file from), an arbitrarily specified mapped drive. Their G:\ may relate to \server4\some\location\therein, but presumably any drive path given to the server via a web form will only send the G:\ file path. Is there a way to determine the ‘real path’ of mapped network drives?

Any solution would be used to stage files from Server 3 to Server 2 when the files are being worked on – the emphasis being on these giant files not having to pass through the user’s local machine first.

Please let me know if you have comments and I will try to make this question more coherant if it is unclear.

Answer 1

As far as I’m aware (and I could be wrong) there is no standard way to determine the UNC path of a mapped drive from a browser.

The only way to do this would be to have some kind of control within the web page. Could be ActiveX or maybe flash. I’ve seen ActiveX doing this, but not flash.

In the past when designing web based systems that need to know the UNC path of a user’s mapped drive I’ve had to have a translation of drive to UNC path stored server side. I did have a luxury though of knowing which drive would map to what UNC path. If the user can set arbitrary paths then this obviously won’t work.

Ok, as I’m procrastinating and avoiding real work I’ve given this some thought.

I’ll preface this by saying that I’m in no way a Linux expert and the system I’m about to describe has just been thought up off the top of my head and is not something you’d want to put into any kind of production. However, it might help you down the right path.

So, you have 3 servers, the Interface Server (LAMP stack I’m assuming?) your Staging Server and your File Store Server. You will also have Client Machines and Network Shares. For the purpose of this design your Network Shares are hosted on nix boxes that your File Store can scp from.

You’d create your frontend website that tracks and stores information about files etc. This will also hold the details about which files are being copied, which are in Staging and so on.

You’ll also need some kind of Service running on the File Store Server. I’ll call this the File Copy Service. This will be responsible for coping the files from your servers hosting the network shares.

Now, you’ve still got an issue with how you figure out what path the users file is actually on. If you can stop users from mapping their own drives and force them to use consistent drive letters then you could keep a translation of drive letter to UNC path on the server. If you can’t, well I’ll let you figure that out. If you’re in a windows domain you can force the drive mappings using Group Policies.

Anyway, the process for the system would work something like this.

1. User goes to system and selects a file
2. The Interface server take the file path and calls the File Copy Service on the File Store Server
3. The File Copy Service connects to the server that hosts the file and initiates the copy. If they’re all nix boxes you could easily use something like SCP. Now, I haven’t actually looked up how to do it but I’d be very surprised if you can’t get a running total of percentage complete from SCP as it’s copying. With this running total the File Copy Service will be updating the database on the Interface Server with how the copy is doing so the user can see this from the Interface Server.

The File Copy Service can also be used to move files from the File Store to the staging server.

As i said very roughly thought out. The above would work, but it all depends a lot on how your systems are set up etc.

Having said all that though, there must be software that would do this out there. Have you looked?