Question

I have a table with about 2,000,000 rows. I need to query one of the columns to retrieve the rows where a string exsists as part of the value.

When I run the query I will know the position of the string, but not before hand. So a view which takes a substring is not an option.

As far as I can see I have three options

1. using like ‘% %’
2. using instr
3. using substr

I do have the option of creating a function based index, if I am nice to the dba.

At the moment all queries are taking about two seconds. Does anyone have experience of which of these options will work best, or if there is another option? The select will be used for deletes every few seconds, it will typically select 10 rows.

edit with some more info

The problem comes about as we are using a table for storing objects with arbitrary keys and values. The objects come from outside our system so we have limited scope to control them so the text column is something like ‘key1=abc,key2=def,keyn=ghi’ I know this is horribly denormalised but as we don’t know what the keys will be (to some extent) it is a reliable way to store and retrieve values. To retrieve a row is fairly fast as we are searching the whole of the column, which is indexed. But the performance is not good if we want to retrieve the rows with key2=def.

We may be able to create a table with columns for the most common keys, but I was wondering if there was a way to improve performance with the existing set up.

Answer 1

In Oracle 10:

CREATE TABLE test (tst_test VARCHAR2(200));  CREATE INDEX ix_re_1 ON test(REGEXP_REPLACE(REGEXP_SUBSTR(tst_test, 'KEY1=[^,]*'), 'KEY1=([^,]*)', '\1'))  SELECT  * FROM    TEST WHERE   REGEXP_REPLACE(REGEXP_SUBSTR(TST_TEST, 'KEY1=[^,]*'), 'KEY1=([^,]*)', '\1') = 'TEST' 

This will use newly selected index.

You will need as many indices as there are KEYs in you data.

Presence of an INDEX, of course, impacts performance, but it depends very little on REGEXP being there:

SQL> CREATE INDEX ix_test ON test (tst_test)   2  / Index created Executed in 0,016 seconds  SQL> INSERT   2  INTO   test (tst_test)   3  SELECT 'KEY1=' || level || ';KEY2=' || (level + 10000)   4  FROM   dual   5  CONNECT BY   6     LEVEL <= 1000000   7  / 1000000 rows inserted Executed in 47,781 seconds  SQL> TRUNCATE TABLE test   2  / Table truncated Executed in 2,546 seconds  SQL> DROP INDEX ix_test   2  / Index dropped Executed in 0 seconds  SQL> CREATE INDEX ix_re_1 ON test(REGEXP_REPLACE(REGEXP_SUBSTR(tst_test, 'KEY1=[^,]*'), 'KEY1=([^,]*)', '\1'))   2  / Index created Executed in 0,015 seconds   SQL> INSERT       2  INTO   test (tst_test)       3  SELECT 'KEY1=' || level || ';KEY2=' || (level + 10000)       4  FROM   dual       5  CONNECT BY       6     LEVEL <= 1000000       7  / 1000000 rows inserted Executed in 53,375 seconds 

As you can see, on my not very fast machine (Core2 4300, 1 Gb RAM) you can insert 20000 records per second to an indexed field, and this rate almost does not depend on type of INDEX being used: plain or function based.