Question

When sending uncontrolled amounts of TextWebSocketFrames to a simple Netty WebSocket echo server implementation (slightly modified version of the implementation found in the examples package), without waiting for a client side sync() on the ChannelFuture, the heap memory usage on the server is growing exponentially until it finally goes out of memory.

Testcase: the client does not wait until the actual bytes have been written, nor does it wait for the server to “echo” the text back to the client before writing the next text frame

for (int i = 0; i < 10000000; i++) {
    ch.write(new TextWebSocketFrame("Message #" + i));
}

When observing the yourkit memory profiler (15 seconds into the test with approx. 20.000 frames written), the number of BigEndianHeapChannelBuffer objects has grown excessively.

BigEndianHeapChannelBuffer 284,509 (objects) 9,104,288 (shallow size) (after ~30.000 frames sent within 10 second window)

On the server side a big pile up can be observed mainly from BigEndianHeapChannelBuffers and CompositeChannelBuffers objects which are never cleaned nor garbage collected (which might not be possible as references are held). I’m guessing this has something to do with the (single) worker thread not being able to write the downstream “echo” response to the client channel because it’s busy processing the rapidly incoming requests from the client

Is there a way to prevent/throttle this (accidental denial of service) on the server side?

Answer 1

In Netty, most I/O operations are asynchronous. Therefore, writing thousands of messages without waiting for the previous writes to be finished will get you an OutOfMemoryError. To avoid that, I prefer having a counter variable to count the number of pending writes. For example:

private final AtomicInteger pendingWrites = new AtomicInteger();

...
while (pendingWrites.get() < MAX_PENDING_WRITES) {
  pendingWrites.incrementAndGet();
  ch.write(msg).addListener(new ChannelFutureListener() {
    ...
    pendingWrites.decrementAndGet();
    if (pendingWrites.get() < MAX_PENDING_WRITES) {
      // resume writing here
    }
  }
}

Alternatively, you can use ChunkedWriteHandler which essentially does the same job.