Question

How does the work_mem option in Postgres work? Here’s the description from http://www.postgresql.org/docs/8.4/static/runtime-config-resource.html:

Specifies the amount of memory to be used by internal
sort operations and hash tables before switching to
temporary disk files. The value defaults to one megabyte
(1MB). Note that for a complex query, several sort or
hash operations might be running in parallel; each one
will be allowed to use as much memory as this value
specifies before it starts to put data into temporary 
files. Also, several running sessions could be doing
such operations concurrently. So the total memory used
could be many times the value of work_mem; it is
necessary to keep this fact in mind when choosing the
value. Sort operations are used for ORDER BY, DISTINCT,
and merge joins. Hash tables are used in hash joins,
hash-based aggregation, and hash-based processing of IN
subqueries. 

I’m probably totally wrong here but..isn’t “switching to temporary disk files” essentially the same thing as “virtual memory” in the operating system? Wouldn’t the OS just create a swap file once the RAM is gone? Wouldn’t it be better to set this to something like 100TB and let the OS figure it out? Before I potentially mess up my system, I want to check if anyone actually tried this approach.

Answer 1

PostgreSQL will for example switch to a sorting operation more suitable for on-disk sort than in-memory sort if it knows the sort will happen on disk – which it won’t know if it happens in swap.

Also, PostgreSQL can switch to a completely different plan (for example, using a different JOIN method) if it figures out the data does not fit in RAM.

Setting work_mem too high will get you a very slow database as soon as you have enough data so that everything doesn’t always fit in RAM anymore.