Question

I’m creating a video sharing website in django.

Currently, if a user registers, and upload a video, it will be uploaded to media/vid/uploaded-vid. Then i converted using ffmpeg to flv. What i would like to do is this:

Someone with the username of alex1 registers, i would like to create a directory for him when he confirms his email, called /media/vid/members-vid/alex1

If he uploaded a video, it will be converted to flv in media/vid/uploaded-vid then copied to /media/vid/members-vid/alex1. And the video in media/vid/uploaded-vid should be deleted.

And I would like to secure /media/vid/. How do you secure django directories? Or is just an apache chmod?

i was wondering if i can use celery/rabbitqm to copy files from one folder to another, or to create new folders..

Answer 1

Your use of a holding directory seems to be a good one. However having directories named after users is risky (what if someone provides a name which is an executable command name?) so using the user id might be better. Indeed if you are going to have lots of users you might want to consider splitting up user directories under some general directories to make finding the directories easier for admin purposes, and because some file systems have rather low limits on the number of entries under a directory — e.g. ext3 allows roughly 32K entries.

For instance, a crude user directory scheme with outer directories named 1 to 9 for subdirectories starting with those numbers will give you a bit more flexibility here.

So, assuming you have your uploaded files in /tmp/upload/1015/, and you want to move them to /var/userdata/01/1015/ and process files in /var/userdata/01/1015/, the following might be a sensible approach:

• /tmp/upload/1015/ will need to be apache user or group writeable
• record the need to process files in /tmp/upload/1015/ in a database, AMQP scheduling service such as RabbitMQ or an RPC/web services call
• a scheduling service client running under its own userid picks up the job from db/AMQP/web call and does the following
  • run a virus check
  • runs a file sanity check to see if the source file is what it expects
  • copies the file to the user folder
  • if successful, removes the original file (or schedules a job to do that if it does not have permission to do so, or relies on a cronjob to delete files in /tmp/upload that are more than 24 hours old)
  • attempts to convert the files, updating relevant success/failure flags in the database as it goes along
  • deletes temporary files/unsuccessful source/output files on completion

The list obviously would go on further for a while, depending on your needs. In the end you will have a user directory with files readable by apache but not writeable by the apache user. If your service grows you can move the video processing components of this service to another server.

By the way there is a very good description of how to do this sort of thing (very simply) by Cal Henderson of Flickr in his O’Reilly book “Building Scalable Web Sites“. It was written in 2006 but his approach is refreshingly direct and straight-forward.