Question

Ok, so I need to create a command that lists the 100 most frequent words in any given file, in a block of text.
What I have at the moment:

$ alias words='tr " " "\012" <hamlet.txt | sort -n | uniq -c | sort -r | head -n 10'

outputs

$ words
     14 the
     14 of
      8 to
      7 and
      5 To
      5 The
      5 And
      5 a
      4 we
      4 that

I need it to output in the following format:

the of to and To The And a we that

((On that note, how would I tell it to print the output in all caps?))

And I need to change it so that I can pipe ‘words’ to any file, so instead of having the file specified within the pipe, the initial input would name the file & the pipe would do the rest.

Answer 1

Okay, taking your points one by one, though not necessarily in order.

---

You can change words to use standard input just by removing the <hamlet.txt bit since tr will take its input from standard input by default. Then, if you want to process a specific file, use:

cat hamlet.txt | words

or:

words <hamlet.txt

---

You can remove the effects of capital letters by making the first part of the pipeline:

tr '[A-Z]' '[a-z]'

which will lower-case your input before doing anything else.

---

Lastly, if you take that entire pipeline (with the suggested modifications above) and then pass it through a few more commands:

| awk '{printf "%s ", $2}END{print ""}'

This prints the second argument of each line (the word) followed by a space, then prints an empty string with terminating newline at the end.

---

For example, the following script words.sh will give you what you need:

tr '[A-Z]' '[a-z]' | tr ' ' '\012' | sort -n | uniq -c | sort -r
    | head -n 3 | awk '{printf "%s ", $2}END{print ""}'

(on one line: I’ve split it for readability) as per the following transcript:

pax> echo One Two two Three three three Four four four four | ./words.sh
four three two 

You can achieve the same end with the following alias:

alias words="tr '[A-Z]' '[a-z]' | tr ' ' '\012' | sort -n | uniq -c | sort -r
    | head -n 3 | awk '{printf \"%s \", \$2}END{print \"\"}'"

(again, one line) but, when things get this complex, I prefer a script, if only to avoid interminable escape characters 🙂