Question

I’m attempting to extract data from log files and organise it systematically. I have about 9 log files which are ~100mb each in size.

What I’m trying to do is: Extract multiple chunks from each log file, and for each chunk extracted, I would like to create a new file and save this extracted data to it. Each chunk has a clear start and end point.

Basically, I have made some progress and am able to extract the data I need, however, I’ve hit a wall in trying to figure out how to create a new file for each matched chunk.

I’m unable to use a programming language like Python or Perl, due to the constraints of my environment. So please excuse the messy command.

My command thus far:

find Logs\ 13Sept/Log_00000000*.log -type f -exec \
sed -n '/LRE Starting chunk/,/LRE Ending chunk/p' {} \; | \
grep -v -A1 -B1 "Starting chunk" > Logs\ 13Sept/Chunks/test.txt

The LRE Starting chunk and LRE Ending chunk are my boundaries. Right now my command works, but it saves all matched chunks to one file (whose size is becoming exessive).

How do I go about creating a new file for each match and add the matched content to it? keeping in mind that each file could hold multiple chunks and is not limited to one chunk per file.

Answer 1

Try something like this:

find Logs\ 13Sept/Log_00000000*.log -type f -print | while read file; do \
sed -n '/LRE Starting chunk/,/LRE Ending chunk/p' "$file" | \
grep -v -A1 -B1 "Starting chunk" > "Logs 13Sept/Chunks/$file.chunk.txt";
done

This loops over the find results and executes for each file and then create one $file.chunk.txt for each of the files.