Question

My EC2 database server failed, preventing SSH or other access (not sure why … grrr AWS … that’s another story).

I was able to make a snapshot of the EBS root volume. I can not boot a new instance from this volume (I’m guessing the boot partition is corrupt). However, I can attach and mount the volume on a new instance.

Now I need to get the PostgreSQL 8.4 on the new machine (Ubuntu 10.04) to load the data from the mounted volume. Is this possible? I’ve tried:

pg_ctl start -D /<mount_dir>/etc/postgresql/8.4/main/

But no joy … PostgreSQL just starts with empty tables.

Is /etc/postgresql/8.4/main/ the correct location for PostgreSQL data files?

Is there a way to recover the data from the mounted volume in a way that PostgreSQL can read again?

Answer 1

(You should really specify your distro and version, etc, with this sort of system admin question.)

Running Pg via pg_ctl as shown above should work, assuming the original database was from Pg 8.4 and so are the binaries you’re trying to use to start it. Perhaps you forgot to stop the instance of PostgreSQL automatically started by the distro? Or connected on the wrong port, so you got the distro’s default instance instead of your DB on another port (or different unix socket path, for unix sockets)?

Personally I wouldn’t do what you’re doing anyway. First, before I did anything else, I’d make a full backup of the entire data directory because you clearly don’t have good backups, otherwise you wouldn’t be worrying about this. Take them now, because if you break something while restoring you’re going to hate yourself. As demonstrated by this fault, trusting Amazon’s storage (snapshot or otherwise) probably isn’t good enough.

Once you’ve done that: The easiest way to restore your DB will be to, on a new instance you know you don’t have any important data on that has the same major version (eg “8.4” or “9.0”) of postgresql as your original instance did installed:

/etc/init.d/postgresql-8.4 stop 
datadir=/var/lib/postgresql/8.4/main
rm -rf "$datadir"
cp -aR /<mount_dir>/etc/postgresql/8.4/main/ "$datadir"
chown -R postgres:postgres "$datadir"
/etc/init.d/postgresql-8.4 start

In other words: take a copy, fix the permissions, start the DB.

You might need to edit /etc/postgresql/8.4/main/postgresql.conf and/or /etc/postgresql/8.4/main/pg_hba.conf because any edits you made to the originals aren’t there anymore; they’re on your corrupted root FS. The postgresql.conf and pg_hba.conf in the datadir are just symlinks to the ones in etc under Debian – something I understand the rationale behind, but don’t love.

Once you get it running, do an immediate pg_dumpall and/or just a pg_dump of your important DB, then copy it somewhere safe.