Question

I just wanted to use grep with option -f FILE. This should make grep use every line of FILE as a pattern and search for it.

Run:

grep -f patternfile searchfile

The pattern-file I used is 400MB large. The file I want to search through is 7GB.
After 3 min the process ended up with 70GB RAM and no reaction.

Is this normal? Am I doing something wrong? Is grep not capable is such large scale?

Thank you for ideas.

Answer 1

You could try breaking the task up such that the grep process ends on each pass of the file. I’m not sure how useful this will be, however, given the sheer size of the file you’re searching.

for pattern in `cat patternFile`
do
    grep "$pattern" searchFile
done