Question

I have the following code where I have collected all the file sizes greater than 40k from my system. I have stored all of this info into a text file. I need to process the file to read the number of times each entry is found in the text file and delete all of those entries.
I have the following code but it does not seem to to working properly.

 #! /bin/sh
  rm -rf /home/b/Desktop/CalcfileSizeGreater40.txt
  filename="/home/b/Desktop/fileSizeGreater40.txt"   
  cat $filename | while read line
    do
  number_of_times=`cat $filename | grep $line | wc -l`
  echo $line:$number_of_times
  echo $line : $number_of_times >> /home/b/Desktop/CalcfileSizeGreater40.txt
  sed '/$line/d' $filename  >tmp
  mv tmp $filename
  done

When I look at the CalcfileSizeGreater40.txt i can see

 131072 : 4
 65553 : 9
 65553 : 9
 65553 : 9
 65553 : 9
 65553 : 9
 65553 : 9
131072 : 4
 65553 : 9
 65553 : 9
 65553 : 9

any ideas as to where I am going wrong ?

Answer 1

You can simplify this line:

number_of_times=`cat $filename | grep $line | wc -l`

to:

number_of_times=$(grep -c "$line" "$filename")

The use of $(...) in place of back-quotes is extra beneficial when you need to nest command execution. You can count occurrences with grep, and you never needed to use cat. It is a good idea to get into the habit of enclosing file names in variables in double quotes just in case the file names end up with spaces in them.

Editing the file that you are using cat on is not a good idea. Because of the way you are operating, the initial cat will echo every line of the original file in turn, completely ignoring any changes you make to a (different) file of the same name with the editing commands. This is why some of your names showed up a lot in the output.

However, what you are basically trying to do is count the number of occurrences of each line in the file. This is conventionally done with:

sort "$filename" |
uniq -c

The sort groups all identical sets of lines together in the file, and uniq -c counts the number of occurrences of each distinct line. It does, however, output the count before the line, so that has to be reversed — we can use sed for that. So, your script could be just:

sizefile="/home/b/Desktop/CalcfileSizeGreater40.txt"
rm -f "$sizefile"
filename="/home/b/Desktop/fileSizeGreater40.txt"
sort "$filename" |
uniq -c |
sed 's/^[     ]*\([0-9][0-9]*\)[     ]\(.*\)/\2 : \1/' > "$sizefile"

I’d be cautious about using rm -fr on your CalcfileSizeGreater40.txt; rm -f is sufficient for a file, and you probably don’t want to remove stuff if it isn’t a file but is a directory.

One pleasant side effect of this is that the code is a lot more efficient than the original as it makes one pass through the file (unless it is so big that sort has to split it up to handle it).

---

I am finding the sed code a little difficult to follow.

I should have explained that the [ ] bits are meant to represent a blank and a tab. On my machine, it appears that uniq only generates spaces, so you could simplify that to:

sed 's/^ *\([0-9][0-9]*\) \(.*\)/\2 : \1/'

The regex looks for the start of a line, any number of blanks, and then a number (which it remembers as \1 because of the \(...\) enclosing it), followed by a space and then ‘everything else’, which is also remembered (as ‘\2’). The replacement then prints the ‘everything else’ followed by a space, colon, space and the count.