Question

I am going to be working with Grep for looking into a very large dataset. Pretty much a lot of logs. I am not sure is it better to grep it once using some sometimes crazy complex regex, or try to do a few simple grep calls and return a smaller and smaller dataset each time. I am not the best at regex but I will be able to store the regex I would need in the log searching tool, I am just looking for which method could be best.

Edit:

Sorry I forgot to add in that info it was in my last post for a different grep question. The files are logs, They could be small or large the one i am testing with is 1.9gb so far So i think that is a pretty decent size to think about, the thing is the logs will start of small and then they will grow and the logs will roate at some point lets say everyday, but If i am doing like my Apache access logs and I grep for how often the home page was loaded, or grep for the page load times so i can make an average.

Answer 1

As your dataset is quite large, it would be preferable to invoke grep only once with the right regex. This would avoid excessive IO overhead.