Question

I have a date and a time field in Postgresql. I am reading it in python and need to sort out things on certain days past certain times.

The steps would basically be like this:

1. Select * from x where date > monthdayyear
2. In that subset, select only those that are > time given for that date
3. AND date2 must be < monthdayyear2 AND time2 must be less than time2 given on that date

I know there are definitely some python ways I could do this, by iterating through results and et cetera. I’m wondering if there is a better way than brute forcing this? I would rather not run multiple queries or have to sort out a lot of extra results in the fetchall() if possible.

Answer 1

If I’ve understood your design, this is really a schema design issue. Instead of:

CREATE TABLE sometable (
    date1 date,
    time1 time,
    date2 date,
    time2 time
);

you generally want:

CREATE TABLE sometable (
    timestamp1 timestamp with time zone,
    timestamp2 timestamp with time zone
);

if you want the timestamp converted automatically to UTC and back to the client’s TimeZone, or timestamp without time zone if you want to store the raw timestamp without timezone conversion.

If an inclusive test is OK, you can write:

SELECT ...
FROM sometable 
WHERE '2012-01-01 11:15 +0800' BETWEEN timestamp1 AND timestamp2;

If you cannot amend your schema, your best bet is something like this:

SELECT ...
FROM sometable
WHERE '2012-01-01 11:15 +0800' BETWEEN (date1 + time1) AND (date2 + time2);

This may have some unexpected quirks when it comes to clients in multiple time zones; you may land up needing to look at the AT TIME ZONE operator.

If you need an exclusive test on one side an/or the other, you can’t use BETWEEN since it’s an a <= x <= b operator. Instead write:

SELECT ...
FROM sometable
WHERE '2012-01-01 11:15 +0800' > (date1 + time1)
  AND '2012-01-01 11:15 +0800' < (date2 + time2);

Automating the schema change

Automating a schema change is possible.

You want to query INFORMATION_SCHEMA or pg_catalog.pg_class and pg_catalog.pg_attribute for tables that have pairs of date and time columns, then generate sets of ALTER TABLE commands to unify them.

Determining what a “pair” is is quite application specific; if you’ve used a consistent naming scheme it should be easy to do with LIKE or ~ operators and/or regexp_matches. You want to produce a set of (tablename, datecolumnname, timecolumnname) tuples.

Once you have that, you can for each (tablename, datecolumnname, timecolumnname) tuple produce the following ALTER TABLE statements, which must be run in a transaction to be safe, and should be tested before use on any data you care about, and where the entries in [brackets] are substitutions:

BEGIN;
ALTER TABLE [tablename] ADD COLUMN [timestampcolumnname] TIMESTAMP WITH TIME ZONE;
--
-- WARNING: This part can lose data; if one of the columns is null and the other one isn't
-- the result is null. You should've had a CHECK constraint preventing that, but probably
-- didn't. You might need to special case that; the `coalesce` and `nullif` functions and
-- the `CASE` clause might be useful if so.
--
UPDATE [tablename] SET [timestampcolumnname] = ([datecolumnname] + [timecolumnname]);
ALTER TABLE [tablename] DROP COLUMN [datecolumnname];
ALTER TABLE [tablename] DROP COLUMN [timecolumnname];
-- Finally, if the originals were NOT NULL:
ALTER TABLE [tablename] ALTER COLUMN [timestampcolumnname] SET NOT NULL;

then check the results and COMMIT if happy. Be aware that an exclusive lock is taken on the table from the first ALTER so nothing else can use the table until you COMMIT or ROLLBACK.

If you’re on a vaguely modern PostgreSQL you can generate the SQL with the format function; on older versions you can use string concatenation (||) and the quote_literal function. Example:

Given the sample data:

CREATE TABLE sometable(date1 date not null, time1 time not null, date2 date not null, time2 time not null);
INSERT INTO sometable(date1,time1,date2,time2) VALUES
('2012-01-01','11:15','2012-02-03','04:00');

CREATE TABLE othertable(somedate date, sometime time);
INSERT INTO othertable(somedate, sometime) VALUES
(NULL, NULL),
(NULL, '11:15'),
('2012-03-08',NULL),
('2014-09-18','23:12');

Here’s a query that generates the input data set. Note that it relies on the naming convention that matching column pairs always have a common name once any date or time word is removed from the column. You could instead use adjacency by testing for c1.attnum + 1 = c2.attnum.

BEGIN;

WITH 
-- Create set of each date/time column along with its table name, oids, and not null flag
cols AS (
    select attrelid, relname, attname, typname, atttypid, attnotnull 
    from pg_attribute 
    inner join pg_class on pg_attribute.attrelid = pg_class.oid 
    inner join pg_type on pg_attribute.atttypid = pg_type.oid 
    where NOT attisdropped AND atttypid IN ('date'::regtype, 'time'::regtype)
),
-- Self join the time and date column set, filtering the left side for only dates and
-- the right side for only times, producing two distinct sets. Then filter for entries
-- where the names are the same after replacing any appearance of the word `date` or
-- `time`.
tableinfo (tablename, datecolumnname, timecolumnname, nonnull, hastimezone) AS (
    SELECT 
        c1.relname, c1.attname, c2.attname, 
        c1.attnotnull AND c2.attnotnull AS nonnull, 
        't'::boolean AS withtimezone
    FROM cols c1 
    INNER JOIN cols c2 ON (
        c1.atttypid = 'date'::regtype 
        AND c2.atttypid = 'time'::regtype 
        AND c1.attrelid = c2.attrelid
        -- Match column pairs; I used name matching, you might use adjancency:
        AND replace(c1.attname,'date','') = replace(c2.attname,'time','')
    )
)
-- Finally, format the results into a series of ALTER TABLE statements.
SELECT format($$
    ALTER TABLE %1$I ADD COLUMN %4$I TIMESTAMP %5$s;
    UPDATE %1$I SET %4$I = (%2$I + %3$I);
    ALTER TABLE %1$I DROP COLUMN %2$I;
    ALTER TABLE %1$I DROP COLUMN %3$I;
$$ || 
    -- Append a clause to make the column NOT NULL now that it's populated, only
    -- if the original date or time were NOT NULL:
    CASE 
       WHEN nonnull
       THEN '    ALTER TABLE %1$I ALTER COLUMN %4$I SET NOT NULL;'
       ELSE ''
    END,

    -- Now the format arguments
    tablename,           -- 1
    datecolumnname,      -- 2
    timecolumnname,      -- 3
    -- You'd use a better column name generator than this simple example:
    datecolumnname||'_'||timecolumnname,  -- 4
    CASE 
       WHEN hastimezone THEN 'WITH TIME ZONE' 
       ELSE 'WITHOUT TIME ZONE' 
    END                  -- 5
)
FROM tableinfo;

You can read the results and send them as SQL commands in a second session, or if you want to get fancy you can write a fairly simple PL/PgSQL function that LOOPs over the results and EXECUTEs each one. The query produces output like:

    ALTER TABLE sometable ADD COLUMN date1_time1 TIMESTAMP WITH TIME ZONE;
    UPDATE sometable SET date1_time1 = (date1 + time1);
    ALTER TABLE sometable DROP COLUMN date1;
    ALTER TABLE sometable DROP COLUMN time1;
    ALTER TABLE sometable ALTER COLUMN date1_time1 SET NOT NULL;

    ALTER TABLE sometable ADD COLUMN date2_time2 TIMESTAMP WITH TIME ZONE;
    UPDATE sometable SET date2_time2 = (date2 + time2);
    ALTER TABLE sometable DROP COLUMN date2;
    ALTER TABLE sometable DROP COLUMN time2;
    ALTER TABLE sometable ALTER COLUMN date2_time2 SET NOT NULL;

    ALTER TABLE othertable ADD COLUMN somedate_sometime TIMESTAMP WITHOUT TIME ZONE;
    UPDATE othertable SET somedate_sometime = (somedate + sometime);
    ALTER TABLE othertable DROP COLUMN somedate;
    ALTER TABLE othertable DROP COLUMN sometime;

I don’t know if there’s any useful way to work out on a per-column basis whether you want WITH TIME ZONE or WITHOUT TIME ZONE. It’s likely you’ll land up just doing it hardcoded, in which case you can just remove that column. I put it in there in case there’s a good way to figure it out in your application.

If you have cases where the time can be null but the date non-null or vice versa, you will need to wrap the date and time in an expression that decide what result to return when null. The nullif and coalesce functions are useful for this, as is CASE. Remember that adding a null and a non-null value produces a null result so you may not need to do anything special.

If you use schemas you may need to further refine the query to use %I substitution of schema name prefixes to disambiguate. If you don’t use schemas (if you don’t know what one is, you don’t) then this doesn’t matter.

Consider adding CHECK constraints enforcing that time1 is less than or equal to time2 where it makes sense in your application once you’ve done this. Also look at exclusion constraints in the documentation.