Question

I have a very large (500mb) csv file and another textfile containing one word per line.

When I find a word in the textfile in the csv in a specific collumn I want to output the line (will redirect output to a file). I had a grep solution using grep -iE “(word1|word2|word3|…|wordn)” but it crashed after using 16gb memory with a segfault and only extracted very few entries.

I tried csvtool but it refuses to produce anything valueable.

The solution must work under linux and optimally not cap on memory.

Answer 1

Use awk. Process the file with one word per line first, save its data in an array and for the second file check the specific column using the expression if (column in array):

awk '
    BEGIN {
        FS = OFS = ",";
    }
    FNR == NR {
        data[ $1 ] = 1;
        next;
    }
    FNR < NR {
        if ( $4 in data ) {
            print $0;
        }
    }
' textfile large_csv_file