Apparently the reason for the BigTable architecture has to do with the difficulty scaling relational databases when you’re dealing with the massive number of servers that Google has to deal with.
But technically speaking what exactly makes it difficult for relational databases to scale?
In the enterprise data centers of large corporations they seem to be able to do this successfully so I’m wondering why it’s not possible to simply do this at a greater order of magnitude in order for it to scale on Google’s servers.
In addition to Mitch’s answer, there’s another facet: Webapps are generally poorly suited to relational databases. Relational databases put emphasis on normalization – essentially, making writes easier, but reads harder (in terms of work done, not necessarially for you). This works very well for OLAP, ad-hoc query type situations, but not so well for webapps, which are generally massively weighted in favor of reads over writes.
The strategy taken by non-relational databases such as Bigtable is the reverse: denormalize, to make reads much easier, at the cost of making writes more expensive.