Question

I’ve been given a database which I can’t handle with my pc, because of little available storage and memory.

The person who gave me this db gave me the following details:

The compressed file is about 15GB, and uncompressed it’s around
85-90GB. It’ll take a similar amount of space once restored, so make
sure the machine that you restore it on has at least 220GB free to be
safe. Ideally, use a machine with at least 8GB RAM – although even our
modest 16GB RAM server can struggle with large queries on the tweet
table.

You’ll need PostgreSQL 8.4 or later, and you’ll need to create a
database to restore into with UTF8 encoding (use -E UTF8 when creating
it from the command-line). If this is a fresh PostgreSQL install, I
highly recommend you tweak the default postgresql.conf settings – use
the pgtune utility (search GitHub) to get some sane defaults for your
hardware. The defaults are extremely conservative, and you’ll see
terrible query performance if you don’t change them.

When I told him that my pc sort of sucks, he suggested me to use an Amazon EC2 instance.

My two issues are:

1. How do I upload the db to an Amazon VM?
2. How do I use it after that?

I’m completely ignorant regarding cloud services and databases as you can see. Any relevant tutorial will be highly appreciated.

Answer 1

If you’re new to cloud hosting, rather than using EC2 directly consider using EnterpriseDB’s cloud options. Details here.

If you want to use EC2 directly, sign up and create an instance.

Choose your preferred Linux distro image. I’m assuming you’ll use Linux on EC2; if you want to use Windows that’s because you probably already know how. Let the new VM provision and boot up, then SSH into it as per the documentation available on Amazon for EC2 and for that particular VM image. Perform any recommended setup for that VM image as per its documentation.

Once you’ve done the recommended setup for that instance, you can install PostgreSQL:

• For Ubuntu, apt-get install postgresql
• For Fedora, yum install postgresql
• For CentOS, use the PGDG yum repository, not the outdated version of PostgreSQL provided.

You can now connect to Pg as the default postgres superuser:

sudo -u postgres psql

and are able to generally use PostgreSQL much the same way you do on any other computer. You’ll probably want to make yourself a user ID and a new database to restore into:

echo "CREATE USER $USER;" | sudo -u postgres psql
echo "CREATE DATABASE thedatabase WITH OWNER $USER" | sudo -u postgres psql

Change “thedatabase” to whatever you want to call your db, of course.

The exact procedure for restoring the dump to your new DB depends on the dump format.

For pg_dump -Fc or PgAdmin-III custom-format dumps:

sudo -u postgres pg_restore --dbname thedatabase thebackupfile

See “man pg_restore” and the online documentation for details on pg_restore.

For plain SQL format dumps you will want to stream the dump through a decompression program then to psql. Since you haven’t said anything about the dump file name or format it’s hard to know what to do. I’ll assume it’s gzip’ed (“.gz” file extension), in which case you’d do something like:

gzip -d thedumpfile.gz | sudo -u postgres psql thedatabase

If its file extension is “.bz2” change gzip to bzip2. If it’s a .zip you’ll want to unzip it then run psql on it using sudo -u postgres psql -f thedumpfilename.

Once restored you can connect to the db with psql thedatabase.