Question

I am using a terminal emulator library to create a terminal and then I use it to send the data entered over serial to a serial device. The library can be seen here.

When I enter data into the terminal a strange series of characters is being sent/received. I think the unicode replacement character gets sent over serial, the serial device doesnt know what it is and returns ~0.

Screenshot of what appears in the terminal when i write “test”:

And the log showing the strings sent and the data received.

I create an EmulatorView, it’s the terminal view. it mentions the diamonds here.

private void sendText(CharSequence text) {
                int n = text.length();
                char c;
                try {
                    for(int i = 0; i < n; i++) {
                        c = text.charAt(i);
                        if (Character.isHighSurrogate(c)) {
                            int codePoint;
                            if (++i < n) {
                                codePoint = Character.toCodePoint(c, text.charAt(i));
                            } else {
                                // Unicode Replacement Glyph, aka white question mark in black diamond.
                                codePoint = '\ufffd';
                            }
                            mapAndSend(codePoint);
                        } else {
                            mapAndSend(c);
                        }
                    }
                } catch (IOException e) {
                    Log.e(TAG, "error writing ", e);
                }
            }

Is there any way to fix this? Can anybody see in the library class why this is happening?, How can I refer to � in java to even parse it out if I wanted to? I can’t say if (!str.contains(“�”) I take it.

When I type in the terminal this is run:

public void write(byte[] bytes, int offset, int count) {


 String str;
try {
    str = new String(bytes, "UTF-8");
      Log.d(TAG, "data received in write: " +str );

      GraphicsTerminalActivity.sendOverSerial(str.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
    Log.d(TAG, "exception" );
    e.printStackTrace();
}

        // appendToEmulator(bytes, 0, bytes.length);

 return;
}

This is what I call to send data. sendData(Byte[] data) is a library method.

public static void sendOverSerial(byte[] data) {
        String str;
        try {
            str = new String(data,"UTF-8");
             if(mSelectedAdapter !=null && data !=null){
                 Log.d(TAG, "send over serial string==== " + str);

                mSelectedAdapter.sendData(str.getBytes("UTF-8"));
                 }
        } catch (UnsupportedEncodingException e) {
            Log.d(TAG, "exception");
            e.printStackTrace();
        }

    }

Once data is sent the reply is received here:

public void onDataReceived(int id, byte[] data) {

        try {
            dataReceived = new String(data, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            Log.d(TAG, "exception");
            e.printStackTrace();
        }

        try {
            dataReceivedByte = dataReceived.getBytes("UTF-8");
        } catch (UnsupportedEncodingException e) {
            Log.d(TAG, "exception");
            e.printStackTrace();
        }
        statusBool = true;
        Log.d(TAG, "in data received " + dataReceived);
        ((MyBAIsWrapper) bis).renew(data);


        runOnUiThread(new Runnable(){

            @Override
            public void run() {

                mSession.appendToEmulator(dataReceivedByte, 0, dataReceivedByte.length);

            }});

    viewHandler.post(updateView);

}

Relevant section of library class where characters are written:

Relevant section of class:

private void sendText(CharSequence text) {
                int n = text.length();
                char c;
                try {
                    for(int i = 0; i < n; i++) {
                        c = text.charAt(i);
                        if (Character.isHighSurrogate(c)) {
                            int codePoint;
                            if (++i < n) {
                                codePoint = Character.toCodePoint(c, text.charAt(i));
                            } else {
                                // Unicode Replacement Glyph, aka white question mark in black diamond.
                                codePoint = '\ufffd';
                            }
                            mapAndSend(codePoint);
                        } else {
                            mapAndSend(c);
                        }
                    }
                } catch (IOException e) {
                    Log.e(TAG, "error writing ", e);
                }
            }

            private void mapAndSend(int c) throws IOException {
                int result = mKeyListener.mapControlChar(c);
                if (result < TermKeyListener.KEYCODE_OFFSET) {
                    mTermSession.write(result);
                } else {
                    mKeyListener.handleKeyCode(result - TermKeyListener.KEYCODE_OFFSET, getKeypadApplicationMode());
                }
                clearSpecialKeyStatus();
            }

Answer 1

I have solved this issue by editing the library I am using. They were using a method which converted a byte to an int, it accepted a codePoint and converted it. So for every key press 4 bytes are used. I changed this so that a byte is used instead of an int. No more extra bytes. Nothing got to do with the encoding format.