Question

Theoretical question :

Let’s say there is one source which knows only how to transmit ASCII chars. (0..127)

And let’s say there is an endpoint which receives these chars .

Can the endpoint decode those chars as utf8 ?

    ascii chars
        ...
        ...
         |
         |
         V

    read as utf ?

Something like this pseudo code :

    var txt="אבג";
    var _bytes=Encoding.ASCII.GetBytes(txt);   <= it wont recognize [א] here
    ...transmit...
    var myUtfString=Encoding.UTF8.GetString(getBytesFromWire();  <= some magic has to be done here

Answer 1

That is possible, but not using UTF8.
UTF8 works by encoding multibyte characters into sequences of bytes that are between 128 and 255.
Your ASCII protocol will not be able to transmit those bytes.

Instead, you need some mechanism to store arbitrary Unicode codepoints or bytes in pure ASCII text:

• You can encode the Unicode text using any encoding to get a stream of (non-ASCII) bytes, then transmit those bytes using Base64 encoding
• You can use the UTF7 encoding to encode Unicode codepoints using pure ASCII characters.
  This will be substantially more space-efficient than Base64 if your text is mostly ASCII.