Question

I am trying to remove parts from a binary file which are between the ANSI strings “stringstart” and “stringend”. Is it possible to do this with sed or perl -pe?

I am thinking about some Regex solution but I don’t know how to write it or how well regex works with binary files.

Answer 1

sed is designed for handling text files rather than binary, though these days, the distinction is generally less significant than it once was. The biggest problem is that text files do not contain zero bytes (bytes with value 0) and binary files do, and many C string processing functions stop at the first zero byte. sed also reads ‘lines’ marked by newline characters. Binary files can end up with long lines as a result. Finally, there’s no guarantee about the relative placement of the string start and end markers relative to newlines. All of these characteristics make sed less suitable for this job than Perl is.

In Perl, I’d be sorely tempted to slurp the file into memory, use an appropriate regex to zap the data from the memory image, and then write the result back to an appropriate location.

perl -e 'local($/); $data = <>; $data =~ s/stringstart(.*?)stringend//gms; print $data'

---

Now tested – test data created using:

#!/usr/bin/env perl
use strict;
use warnings;

sub full_set
{
    foreach my $i (0..255) { printf "%c", $i; }
}
sub random_set
{
    my($n) = @_;
    foreach my $i (0..$n) { printf "%c", int(rand(255)); }
}

full_set;
random_set(1024);
printf("stringstart");
full_set;
random_set(512);
full_set;
printf("stringend");
random_set(256);

The script deletes 1045 characters from the input – which corresponds to ‘stringstart’, ‘stringend’ (20) + 2 * 256 + 513 (since random_set(512) prints 513 characters).

Note that the main script will read all files into memory at once. If you want it to process one file at a time, you’ll have to work a little harder; it probably ceases to be a one-liner.