Question

I’ve been noticing some problems with simple aggregate performance in Postgres (8.3). The issue is that if I have a table (say 200M rows) that is unique by (customer_id,order_id), then the query select customer_id,max(order_id) from larger_table group by customer_id is more than an order of magnitude slower than a simple Java/JDBC program that does the following:

1) Initialize an empty HashMap customerMap (this will map id -> max order size)
2) executes “select customer_id,order_id from larger_table”, and gets a streaming result set
3) iterates over the result set, at every row doing something like the following:

long id = resultSet.getLong("customer_id");
long order = resultSet.getLong("order_id");
if (!customerMap.containsKey(id)) 
    customerMap.put(id,order);
else 
    customerMap.put(id,Math.max(order,customerMap.get(id)));

Is this performance difference expected? I should think not, since I imagine the above is pretty close to what is happening internally. Is it evidence that there is something wrong/incorrectly tuned with the db?

Answer 1

It’s probably your work_mem setting being too low. I’d check that first. I have been bitten by this recently. Second most likely problem is that you’re missing a foreign key index.

Exposition follows.

In general, there are several questions that need to be asked whenever database performance looks sub-par:

1. Are you using an up-to-date version? Every point release between 7.4 and 9.0 brought substantial performance improvements with it—if upgrading is possible, it is recommended.
2. Are you running your benchmark on realistic data? PostgreSQL’s query planner will produce different plans on the same table with different data or different quantities of data in it. Make sure you always test with realistic data.
3. What is your PostgreSQL configuration like? The work_mem setting is low out of the box, I myself have run into situations involving GROUP BY where it was artificially choosing the wrong plan because it simply didn’t think it had enough working memory to sort the results in.
4. Is your Java code running on the same machine as your database? If not, you may be seeing differences between computers rather than differences between approaches.
5. Are you missing an index? PostgreSQL doesn’t create indexes automatically for foreign keys, just primary keys. I have also been bitten by this, but if you google around you can find a script that will detect and add missing foreign key indices.

Without examining the query plan it’s not a good idea to second-guess what implementation strategy PostgreSQL has chosen for a given query.