Question

I have a file with contents

abc
def
high
lmn
...
...

There are more than 2 million lines in the files.
I want to randomly sample lines from the files and output 50K lines. Any thoughts on how to approach this problem? I was thinking along the lines of Perl and its rand function (Or a handy shell command would be neat).

Related (Possibly Duplicate) Questions:

• Randomly Pick Lines From a File Without Slurping It With Unix
• How can I get exactly n random lines from a file with Perl?

Answer 1

Assuming you basically want to output about 2.5% of all lines, this would do:

print if 0.025 > rand while <$input>;