Question

I have a user table with a text_pattern_ops index on the column key. The problem is that the data in the key column has underscores within it which need to be escaped. There are two ways (that I know of) to escape underscore, and only in one of these the index is actually used. Can anyone explain why this is so?

I have pasted the results of explain analyse for both the queries below.

Query 1:

EXPLAIN ANALYZE
select distinct userid from user
where userstatus IN ('Active')
and ( key like E'999999999_434153_%' or parentid = 434153) ;

QUERY PLAN:

HashAggregate  (cost=340685.17..340687.84 rows=267 width=4) (actual time=22678.760..22678.760 rows=0 loops=1)
  ->  Seq Scan on user  (cost=0.00..340684.50 rows=267 width=4) (actual time=22678.754..22678.754 rows=0 loops=1)
        Filter: (((userstatus)::text = 'Active'::text) AND (((key)::text ~~ '999999999_434153_%'::text) OR (parentid = 434153)))
Total runtime: 22678.879 ms

Query 2:

EXPLAIN ANALYZE
select distinct userid from user
where userstatus IN ('Active')
and ( key like '999999999\\_434153\\_%' or parentid = 434153) ;

Produces a warning:

WARNING:  nonstandard use of \\ in a string literal
LINE 1: ...userstatus IN ('Active') and ( key like '999999999...
                                                             ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.

QUERY PLAN:

HashAggregate  (cost=344.50..347.17 rows=267 width=4) (actual time=226.127..226.127 rows=0 loops=1)
  ->  Bitmap Heap Scan on user  (cost=11.09..343.83 rows=267 width=4) (actual time=226.123..226.123 rows=0 loops=1)
        Recheck Cond: (((key)::text ~~ '999999999\\_434153\\_%'::text) OR (parentid = 434153))
        Filter: (((userstatus)::text = 'Active'::text) AND (((key)::text ~~ '999999999\\_434153\\_%'::text) OR (parentid = 434153)))
        ->  BitmapOr  (cost=11.09..11.09 rows=84 width=0) (actual time=226.121..226.121 rows=0 loops=1)
              ->  Bitmap Index Scan on user_key_idx  (cost=0.00..5.44 rows=1 width=0) (actual time=145.758..145.758 rows=0 loops=1)
                    Index Cond: (((key)::text ~>=~ '999999999_434153_'::text) AND ((key)::text ~<~ '999999999_434153`'::text))
              ->  Bitmap Index Scan on user_parentid_key1  (cost=0.00..5.52 rows=84 width=0) (actual time=80.358..80.358 rows=0 loops=1)
                    Index Cond: (parentid = 434153)
Total runtime: 226.256 ms

Answer 1

You are confusing two levels of escaping.

1. Posix-style escaped strings E'foo'. Check your setting for standard_conforming_strings.

2. Pattern for LIKE operator, where _ has a special meaning that can be escaped. I quote the manual:

To match a literal underscore or percent sign without matching other
characters, the respective character in pattern must be preceded by
the escape character. The default escape character is the backslash
but a different one can be selected by using the ESCAPE clause. To
match the escape character itself, write two escape characters.

The index can only be used for left anchored patterns. If you have an underscore (_) in the middle of the pattern, the index cannot be used. Like in this pattern expression:

key like E'999999999_434153_%'

Unescaped _ in the middle of the pattern, wildcard for any single character – may not be able to use a B-tree index with text_pattern_ops, especially in older versions. Also see @Richard’s comment.

In this pattern the _ is escaped, which means it stands for a literal _ and not as wildcard for a single character -> index not used.

key like '999999999\\_434153\\_%'

Assuming you have standard_conforming_strings = OFF. With standard_conforming_strings = ON this would result in a pattern looking for a literal \ and a wildcard _ which might not use the index either.

You may be interested in the additional module pg_trgm, which allows GiST or GIN indexes to support any LIKE expression. See:

• Pattern matching with LIKE, SIMILAR TO or regular expressions
• How is LIKE implemented?