Question

I have a c++ program which generates many data files each of them containing three columns. Now in each of these data files there can be some anomalous entries where the third column would have -nan. How do I write a script to so that it opens each of these data files and finds all the rows in which the third column has nan and deletes all such rows?
Is it possible to write a script in bash or python that can do this?
Example:

100   0.1    15.8334
100   0.2    16.7895
100   0.3     -nan
100   0.4    15.8543
100   0.5      -nan

In this file I would like the 3rd and the 5th rows to be deleted so that my file looks like

100   0.1    15.8334
100   0.2    16.7895
100   0.4    15.8543

Answer 1

sed -i -e '/-nan/d' datafile.txt

To operate on multiple files, you can replace “datafile.txt” with a glob that matches all the files, or use a for loop

for file in data1.txt data2.txt data3.txt; do
    sed -i -e '/-nan/d' $file
done

or perhaps the find command:

find . -name "data*.txt" -exec sed -i -e '/-nan/d' {} +