Question

I need a fast way to match a value in AWK, I have 250k values to search.

I’m doing something like this:

    #list with 250k numbers instead of four
    number_list="9998532001 9998536052 9998543213 9998544904"

    if ( index(number_list,substr($5,9)) ) 
         {printf "Value: %s\n",$5;}

Any suggestions for a faster search ?

Answer 1

If the substring you are searching for is of a consistent length and position in the target string (say the last 6 digits), then you could preprocess the list into an array and you’d be good to go.

Preprocessing step (perhaps in the BEGIN target)

n=split(numbers_list,a," "); # Rip in input sting into pieces
for ( num in a ) {
    key=substr(a[num],length(a[num])-6,6);  # Get the last six digits

    # Error processing (i.e. collision handling) should go here 

    list[key]=a[num];
}

Then when you need to do the lookup

i=list[substr($5,9)]   # i is now the full number associated with the key

This is only a win if you will do many lookups, because you still have to pay the cost of iterating through that whole list (twice, in fact) during the pre-processing.

---

Note that exact matching to the whole number qualifies as a substring of known length and position, just use key=a[num] (which looks funny and leads to several simplifications of the above code, but I’m sure you can figure it out).

---

If you are looking for any occurrence of substring($5,9) in any of the numbers, this won’t work, you’ll have to iterate through n every time.