Question

Problem Statement:-

I need to search a particular String Pattern in around 10000 files and find the records in the files which contains that particular pattern. I can use grep here, but it is taking lots of time.

Below is the command I am using to search a particular string pattern after unzipping the dat.gz file

gzcat /data/newfolder/real-time-newdata/*_20120809_0_*.gz | grep 'b295ed051380a47a2f65fb75ff0d7aa7^]3^]-1'

If I simply count how many files are there after unzipping the above dat.gz file

gzcat /data/newfolder/real-time-newdata/*_20120809_0_*.gz | wc -l

I get around 10000 files. And I need to search the above string pattern in all these 10000 files and find out the records which contains the above String Pattern. And my above command is working fine but it is very very slow.

What is the best approach on this? Should we take 100 files at a time and search for the particular String Pattern in that 100 files parallelly.

Note:

I am running SunOS

bash-3.00$ uname -a
SunOS lvsaishdc3in0001 5.10 Generic_142901-02 i86pc i386 i86pc

Answer 1

Do NOT run this in parallel!!!! That’s going to bounce the disk head all over the place, it will be much slower.

Since you are reading an archive file there’s one way to get a substantial performance boost–don’t write the results of the decompression out. The ideal answer would be to decompress to a stream in memory, if that’s not viable then decompress to a ramdisk.

In any case you do want some parallelism here–one thread should be obtaining the data and then handing it off to another that does the search. That way you will either be waiting on the disk or on the core doing the decompressing, you won’t waste any of that time doing the search.

(Note that in case of the ramdisk you will want to aggressively read the files it wrote and then kill them so the ramdisk doesn’t fill up.)