Question

I want to extract rows from a file that match a particular pattern and I want to do this for over 500 files. It should have the ability to retain the unique name of the file as well.

I used awk but then i have to do each file individually.

c:\>gawk "/S1901/" Census_Tract_*.csv > Census_Tract_*.csv

In the example shown in the link here (http://bit.ly/nMX8qh) I want to retain only those records that have S1901 in them. Apologies for the external link but i am not able to retain formatting of the table.

I found some perl code that I used to write it but it retains all the rows and does not select only those rows/records where the pattern matches. Any tips would be much appreciated. The perl code is below:

#perl -w
$pattern = "Subject_Census*.csv";  # process only those files that match pattern
while (defined ($in = glob($pattern))) {
  ($out = $in) =~ s/\.csv$/.outcsv/;            # read from "xyz.in" and write to "xyz.out"
  open (IN, "<", $in) or die "Can't open $in for reading: $!";
  open (OUT,">>", $out) or die "Can't open $out for writing: $!";
  while (<IN>) {
    $mystring  =~ /S1901/;
    print OUT $_ if $mystring == 0;
  }
  close (IN) or die "Can't close $in: $!";  # good idea to do some housekeeping
  close (OUT) or die "Can't close $out: $!";
}

Answer 1

Untested:

use strict;
use warnings;
use autodie;

my $files_list_filename = 'files.txt';

open my $fl, '<', $files_list_filename;
my @list_of_files = <$fl>;
chomp @list_of_files;
close $fl;

foreach my $file ( @list_of_files ) {
    open my $test_fh, '<', $file;
    while ( my $line = <$test_fh> ) {
        if( $line =~ m/S1901/ ) {
            print "$file at $.: $line";
        }
    }
    close $test_fh;
}

Is that sort of what you had in mind? It opens a file named filelist.txt and reads in a list of however many filenames you want to give it. Then it iterates over that list, opening each file one by one, scanning each file one by one, and if a line is found containing the trigger text, it prints the filename and line number, as well as the line itself where the trigger was met. Then it moves on to the next.