Question

I’ve got a simple socket server (it is for HL7 communication). When it runs longer in production, socket threads hang and consume a lot of CPU time.

This is the relevant code (shortened) for the listener thread:

public void run() {
    try {
        serverSocket = new ServerSocket(port, backlog, bindAddress);
        serverSocket.setSoTimeout(timeout); // 1000 ms
        do {
            Socket socket = null;
            try {
                socket = serverSocket.accept();
            } catch (SocketTimeoutException to) {
                socket = null;
            } catch (InterruptedIOException io) {
                socket = null;
            } catch (IOException e) {
                logger.fatal("IO exception while socket accept", e);
                socket = null;
            }

            try {
                if (socket != null)
                    processConnection(socket);
            } catch (RuntimeException e) {
                logger.fatal("caught RuntimeException trying to terminate listener thread", e);
            }
        } while (running);
    } catch (IOException e) {
        logger.fatal("error binding server socket - listener thread stopped", e);
    }
}

This code starts a new thread for processing an incoming connection:

protected void processConnection(Socket socket) {
    Hl7RequestHandler requestHandler = createRequestHandler();
    requestHandler.setSocket(socket);
    requestHandler.start();
}

This is the code for the request handler thread (keepAlive is set to true):

public void run() {
    try {
        setName("Hl7RequestHandler-" + socket.getPort());
        processRequest();
    } catch (IOException e) {
        logger.fatal("IO exception during socket communication", e);
    }
}

public void processRequest() 
throws IOException {
    socket.setSoTimeout(socketTimeout); // 1000 ms

    InputStream inputStream = socket.getInputStream();
    OutputStream outputStream = socket.getOutputStream();

    BufferedReader inputReader = new BufferedReader(new InputStreamReader(inputStream, encoding));
    Writer outputWriter = new OutputStreamWriter(outputStream, encoding);

    int timeouts = 0;
    boolean failure = false;
    do {
        StringBuilder message = new StringBuilder();
        try {
            char c;
            do {
                c = (char)inputReader.read();
                if ((c == CARRIAGE_RETURN || c == START_OF_MESSAGE) &&
                    message.length() == 0)
                else if (c != END_OF_MESSAGE && ((short)c) != -1)
                    // ein Byte "Nutzlast"
                    message.append(c);
            } while (c != END_OF_MESSAGE && ((short)c) != -1);
        } catch (SocketTimeoutException te) {
            timeouts++;
            if(!keepAlive && timeouts >= 3 ) {
                socket.close();
                return;
            }
        }

        String messageStr = message.toString();
        if (messageStr.length() == 0)
            continue;

        failure = !processMessage(messageStr, outputWriter);
        outputWriter.flush();
        outputStream.flush();

        // nächste Runde?
        if (!keepAlive || failure)
            socket.close();
    } while (keepAlive && !failure);
}

When I test this locally, it works well.

But in production, there are multiple request handler threads that “hang”. “Keep Alive” is ment to hold open the connection waiting for more messages. (To avoid opening up new connections all the time.) I assume inputReader.read() returns -1 after the timeout of 1s, which results in calling the method just again. Why does this eat up all the CPU time?

Have you got any advice?

Thanks in advance,
Matthias

Answer 1

One thing I can see straight off is this :

         char c;
        do {
            c = (char)inputReader.read();
            if ((c == CARRIAGE_RETURN || c == START_OF_MESSAGE) &&
                message.length() == 0)
            else if (c != END_OF_MESSAGE && ((short)c) != -1)
                // ein Byte "Nutzlast"
                message.append(c);
        } while (c != END_OF_MESSAGE && ((short)c) != -1);

is the cast of the inputReader.read() to char. BufferedReader.read() returns an int, a signed value. You cast it to char that is an unsigned value, discarding the negative sign if there is one, a narrowing conversion. Then converting to short is not bringing back the negative sign if there was one.
Try rewriting as :

         char c;
         int val;
        do {
            val = inputReader.read();
            // do this if you want, you don't have to
            c = (char) val;
            if ((c == CARRIAGE_RETURN || c == START_OF_MESSAGE) &&
                message.length() == 0)
            else if (c != END_OF_MESSAGE && ((short)c) != -1)
                // ein Byte "Nutzlast"
                message.append(c);
        } while (c != END_OF_MESSAGE && val != -1);

I’ve taken another look at your loop and I’m confused.

        char c;
        do {
            c = (char)inputReader.read();
            if ((c == CARRIAGE_RETURN || c == START_OF_MESSAGE) &&
                message.length() == 0)
            else if (c != END_OF_MESSAGE && ((short)c) != -1)
                // ein Byte "Nutzlast"
                message.append(c);
        } while (c != END_OF_MESSAGE && ((short)c) != -1);

The logic of your if statements are confusing (to me at least).
You have no statements for the first if clause, not even an empty statement.
You have to have either {} or a ; Does your code compile?