Question

How does a large video site like YouTube or DailyMotion handle a large amount of simultaneous video uploads. For example, to be able to handle the bandwidth from 1000s of users, what special considerations need to be made in web servers, hardware, etc.? Thank you.

Answer 1

This is what is known as the c10k problem — how do you handle 10,000 clients simultaneously.

The web server software must be well-written, so each server can talk with hundreds or thousands of clients at once.

The storage backend must be well-designed, so each server has relatively uncontested write ability to the storage network. The storage network must be redundant, so dead drives don’t destroy user data. (Even a failure rate of 0.01% / year means hundreds of dead drives a year when you’ve got millions of them.)

Database backends must be prepared to scale outwards incredibly — nothing silly like “lock table” when inserting new records.

The routing framework must be prepared to route traffic to nearby datacenters, based on the actual traffic cost of the ISPs involved. (No sense bringing the whole network to its knees with video uploads crossing in the pipes on their way to datacenters.)

The routing framework must be prepared to handle entire sites going offline at once, to redirect failed uploads to new server centers without too much pain.

The data centers must be built to handle power failures, air conditioning failures, network failures, breaches by malicious or curious members of the public trying to get in, typically by having two or more power sources, including massive site-wide uninterruptible power supplies and generators; multiple independent air conditioning that can run the entire site; two, three, four, or more network sources; physical security measures and potentially armed guards.

Hurricane Electric has some nice photos up of some of their data centers. They look impressive but are a tiny fraction of Google, Facebook, or Amazon’s datacenter needs.