Question

I have a processing in database that iterate over an entire table. At some point, I need to check if a VARCHAR matches any value in a column.

I want to know what is the fast way to do this. Do SQL comparison each time? Retrieve the table values to a VARCHAR array? Another solution?

The iteration in the table is over millions, so the comparison will be done million times. In the match table, there a few hundred values.

So, what is the best approach for this?

Answer 1

Have an index on the varchar column in the small table. Then the look-up will be as fast as possible. A plain btree index (default) is good for the equality operator.

CREATE INDEX smalltbl_lookup ON smalltbl(lookup);

If you only look up some values, this will be the fastest way:

EXISTS (SELECT * FROM smalltbl WHERE lookup = bigtbl.lookup)

Resulting in an index search. If you look up all values (does not sound like you do), a LEFT JOIN is faster:

SELECT *
FROM bigtbl b
LEFT JOIN smalltbl s USING (lookup)

Transforming the values from the small table into array and checking there cannot compete with an index search.