Question

I’m trying to use sed to drop strings containing repeated characters before appending them to a file.
So far I have this, to drop stings with consecutive repetition like ‘AA’ or ’22’, but I’m struggling with full string repetition and incremental characters.

generic string generator | sed '/\([^A-Za-z0-9_]\|[A-Za-z0-9]\)\1\{1,\}/d' >> parsed sting to file

I also want to drop strings contain any repetition like ‘ABA’.
As well as, strings containing any ascending or descending characters like ‘AEF’ or ‘AFE’.

I’m assuming it would be easier to use multiple passes of sed to drop the unwanted strings.

** A little more information to try to avoid the XY problem mentioned. **

The character strings could be from 8 to 64 in length, but in this instance I’m focusing on 8. While at the same time I’ve restricted the string generation to only output an upper-case alpha string (A-Z). This is for a few reasons, but mainly that I don’t want the generated file to have a ridiculously huge footprint.

With the first pass of sed dropping unnecessary outputs like ‘AAAAAAAA’ and ‘AAAAAAAB’ from the stream. This results in the file starting with strings ‘ABABABAB’ and ‘ABABABAC’.

Next pass I want to check that from one character to the next doesn’t increase or decrease by a value of one. So strings like ‘ABABABAB’ would be dropped, but ‘ACACACAC’ would parse to the stream.

Next pass I want to drop strings that contain any repeated characters in the whole string. So strings like ‘ACACACAC’ would be dropped, but ‘ACEBDFHJ’ would parse to the file.

Hope that helps.

Answer 1

In order to do what you’re describing with sed, you’d need to run it many times. Since sed doesn’t understand the concept of “this character is incremental from this other character”, you need to run it across all possible combinations:

sed '/AB/d'
sed '/BC/d'
sed '/CD/d'
sed '/DE/d'

etc.

For descending characters, the same thing:

sed '/BA/d'
sed '/CB/d'

In order to then drop strings with repeated characters, you can do something like this:

sed '/\(.\).*\1/d'

The following should do the trick:

generic string generator |sed '/\(.\).*\1/d'|sed /BA/d|sed /AB/d||sed /CB/d|sed /BC/d|sed /DC/d|sed /CD/d|sed /ED/d|sed /DE/d|sed /FE/d|sed /EF/d|sed /GF/d|sed /FG/d|sed /HG/d|sed /GH/d|sed /IH/d|sed /HI/d|sed /JI/d|sed /IJ/d|sed /KJ/d|sed /JK/d|sed /LK/d|sed /KL/d|sed /ML/d|sed /LM/d|sed /NM/d|sed /MN/d|sed /ON/d|sed /NO/d|sed /PO/d|sed /OP/d|sed /QP/d|sed /PQ/d|sed /RQ/d|sed /QR/d|sed /SR/d|sed /RS/d|sed /TS/d|sed /ST/d|sed /UT/d|sed /TU/d|sed /VU/d|sed /UV/d|sed /WV/d|sed /VW/d|sed /XW/d|sed /WX/d|sed /YX/d|sed /XY/d|sed /ZY/d|sed /YZ/d

I only tested this on a few input samples, but they all seemed to work.

Note that this is quite ungainly, and would be better done by something a little more sophisticated than sed. Here’s a sample in python:

import math
def isvalid(x):
   if set(len(x)) < len(x):
     return False
   for a in range(1, len(x)):
     if math.fabs(ord(x[a])-ord(x[a-1])) == 1:
       return False
   return True

This is much more readable than the giant set of sed calls, and has the same functionality.