Question

I have a very basic understanding of shell scripting, but what I need to do requires more complex commands.

For one task, I need to find and replace html code within the index.html files on my server. These files are in multiple directories with a consistent naming convention. ([letter][3-digit number]) See the example below.

files: index.html
path: /www/mysite/board/today/[rsh][0-9]/ 
string to find: (div id="id")[code](/div)<--#include="(path)"-->(div id="id")[more code](/div)
string to replace with: (div id="id")<--include="(path)"-->(/div)

I hope you don’t mind the pseudo-regex. The folders containing my target index.html files look similar to r099, s017, h123. And suffice the say, the html code I’m trying to replace is relatively long, but its still just a string.

The second task is similar to the first, only the filename changes as well.

files: [rsh][0-9].html
path: www/mysite/person/[0-9]/[0-9]/[0-9]/card/2011/
string: (div id="id")[code](/div)<--include="(path)"-->(div id="id")[more code](/div)
string to replace with: (div id="id")<--include="(path)"-->(/div)

I’ve seen other examples on SO and elsewhere on the net that simply show scripts modifying files under a single directory to find & replace a string without any special characters, but I haven’t seen an example similar to what I’m trying to do just yet.

Any assistance would be greatly appreciated.

Thank You.

Answer 1

You have three separate sub-problems:

1. replacing text in a file
2. coping with special characters
3. selecting files to apply the transformation to

​1. The canonical text replacement tool is sed:

sed -e 's/PATTERN/REPLACEMENT/g' <INPUT_FILE >OUTPUT_FILE

If you have GNU sed (e.g. on Linux or Cygwin), pass -i to transform the file in place. You can act on more than one file in the same command line.

sed -i -e 's/PATTERN/REPLACEMENT/g' FILE OTHER_FILE…

If your sed doesn’t have the -i option, you need to write to a different file and move that into place afterwards. (This is what GNU sed does behind the scenes.)

sed -e 's/PATTERN/REPLACEMENT/g' <FILE >FILE.tmp
mv FILE.tmp FILE

​2. If you want to replace a literal string by a literal string, you need to prefix all special characters by a backslash. For sed patterns, the special characters are .\[^$* plus the separator for the s command (usually /). For sed replacement text, the special characters are \& and newlines. You can use sed to turn a string into a suitable pattern or replacement text.

pattern=$(printf %s "$string_to_replace" | sed -e 's![.\[^$*/]!\\&!g')
replacement=$(printf %s "$replacement_string" | sed -e 's![\&]!\\&!g')

​3. To act on multiple files directly in one or more directories, use shell wildcards. Your requirements don’t seem completely consistent; I think these are the patterns you’re looking for, but be sure to review them.

/www/mysite/board/today/[rsh][0-9][0-9][0-9]/index.html
/www/mysite/person/[0-9]/[0-9]/[0-9]/card/2011/[rsh][0-9].html

This will match files like /www/mysite/board/today/r012/index.html and /www/mysite/person/4/5/6/card/2011/h7.html, but not /www/mysite/board/today/subdir/s012/index.html or /www/mysite/board/today/r1234/index.html.

If you need to act on files in subdirectories recursively, use find. It doesn’t seem to be in your requirements and this answer is long enough already, so I’ll stop here.

​4. Putting it all together:

string_to_replace='(div id="id")[code](/div)<--#include="(path)"-->(div id="id")[more code](/div)'
replacement_string='(div id="id")<--include="(path)"-->(/div)'
pattern=$(printf %s "$string_to_replace" | sed -e 's![.\[^$*/]!\\&!g')
replacement=$(printf %s "$replacement_string" | sed -e 's![\&]!\\&!g')
sed -i -e "s/$pattern/$replacement/g" \
  /www/mysite/board/today/[rsh][0-9][0-9][0-9]/index.html \
  /www/mysite/person/[0-9]/[0-9]/[0-9]/card/2011/[rsh][0-9].html

Final note: you seem to be working on HTML with regular expressions. That’s often not a good idea.