Question

In an attempt to optimise my bash script I loaded my file into an array and tried to grep from there, and I notice that this grep from memory is much, much slower than a standard grep from file, even taking into account the fact that the disk I/O is taking out of the equation.

1) OK so I have a large file (about 3000 lines) with name=value pairs, this is my “cache”. I load it into an array from file (straight-forward enough)

# load to array
l_i_array_index=1
while read line
  do
  g_a_cache[$l_i_array_index]=$line
  let "l_i_array_index += 1"
  done < $g_f_cache

2) Then I run a little benchmark for the search performance:

time for i in `seq 1 100`
  do
  for l_i_array_index in `seq 1 ${#g_a_cache[@]}`
    do
      echo ${g_a_cache[$l_i_array_index]}
    done | grep -c $l_s_search_string > /dev/null
  done

real    0m14.387s
user    0m13.846s
sys     0m1.781s

3) The same, but directly from disk file:

time for i in `seq 1 100`
  do
  grep -c $l_s_search_string $g_f_cache > /dev/null
  done
real    0m0.347s
user    0m0.161s
sys     0m0.136s

So performance is 13 to 40 times worse, when it should be better.

My question is: 1) what is the reason for this strange behaviour 2) is this solvable within bash or should I bite the bullet and finally redo it in Python

P.S. The test is done on Mac (bash v4), in Cygwin, time per ONE search is over a second with normal grep (which is faster), and over 10s with the array method. The script is close to unusable..

Answer 1

The grep program has been heavily optimized over many, many years by top experts in the field of searching and algorithm design. You’re simply not going to beat it with a shell script. That’s an absurd notion.

I honestly can’t imagine why you would expect to be anywhere near as fast as grep. Perhaps you’re thinking that all of greps disk I/O actually requires doing something to the actual physical disk. But that’s not so. Every modern operating system has a disk cache, and the file will be in the cache after the first time it’s read, which takes a tiny fraction of a second.