It seems like bash, and also dash, filter out any ASCII NUL from my scripts.
$ printf 'test="\000a" ; echo ${#test}' | sh
1
$ printf 'test="\001a" ; echo ${#test}' | sh
2
$ printf 'ec\000ho test' | sh
test
$ # (Same for bash)
While I agree that using NUL is a bad bad idea (for example argument passing to programs works with NUL-terminated strings), I don’t see where this behaviour is sanctioned by the POSIX standard.
It gets even worse when this behaviour is deciding on the syntactical correctness of the file.
$ printf 'echo "\\\000"' | sh
sh: Syntax error: Unterminated quoted string
$ printf 'echo "\\\000"' | bash
bash: line 1: unexpected EOF while looking for matching `"'
bash: line 2: syntax error: unexpected end of file
$ printf 'echo "\\\134"' | sh
\
What essential part did I miss, or is the NUL-removal only a decision on how to cope with unspecified behaviour?
The INPUT FILES section in the standard for sh states:
The input file shall be a text file, except that line lengths shall be unlimited. If the input file is empty or consists solely of blank lines or comments, or both, sh shall exit with a zero exit status.
The term “text file” is defined in section 3.395 here as:
A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify “text files” in their STDIN or INPUT FILES sections
If the input is not a text file (which it is not if it contains zero bytes), the behavior is neither meaningful nor predictable.