Question

I’ve got a Postgres 9.1 database that contains weather information. The dataset consists of approximately 3.1 million rows.

It takes about 2 minutes to load the data from a CSV file, and a little less to create a multicolumn index.

Every 6 hours I need to completely refresh the dataset. My current thinking is I would import the new dataset into a different database name, such as “weather_imported” and once the import and index creation are finished, I would drop the original database and rename the imported database.

In theory, clients would continue to query the database during this operation, though if that has ill effects, I could probably arrange to have the clients silently ignore a few errors.

Questions:

1. Will that strategy work?

2. If a client happened to be in the
   process of running a query at the time of DB drop, my assumption is
   the database would not complete the drop until the query were
   finished – true?

3. What if a query happened between the time the
   DB were dropped and the rename? I assume a “database not found”
   error.

4. Is there a better strategy?

Answer 1

Consider the following strategy as an alternative:

• Include a “dataset version” field in the primary table.
• Store the “current dataset version” in some central location, and write your selects to only search for rows which have the current dataset version.
• To update the dataset:
  1. Insert all the data with a new dataset version. (You could just use the start time of the update job as a version.)
  2. Update the “current dataset version” atomically to the value you just inserted.
  3. Delete all data with an older version than the version number you just inserted.

Presto — no need to shuffle databases around.