Question

I am working on a Netty server, I am having issues with a custom handler I created to receive file uploads via HTTP PUT requests. Everything seems to work fine when I just send a few files at a time, however after about 300 connections the server seems to “break”. The server will then throw the follow exception on each received request. After this starts happening, the server no longer handles the requests and needs to be restarted:

    java.lang.IllegalStateException: cannot send more responses than requests
        at org.jboss.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:104)
        at org.jboss.netty.handler.execution.ExecutionHandler.handleDownstream(ExecutionHandler.java:165)
        at org.jboss.netty.channel.Channels.write(Channels.java:605)
        at org.jboss.netty.channel.Channels.write(Channels.java:572)
....

Here is my handler source channelRecieved, all the requests i’m handling are chunked, so I will include those methods below:

@Override
public void messageReceived(ChannelHandlerContext context, MessageEvent event) throws Exception {
    try {
        log.trace("Message recieved");
        if (newMessage) {
            log.trace("New message");
            HttpRequest request = (HttpRequest) event.getMessage();
            setDestinationFile(context, request);
            newMessage = false;
            if (request.isChunked()) {
                log.trace("Chunked request, set readingChunks true and create byte buffer");
                requestContentStream = new ByteArrayOutputStream();
                readingChunks = true;
                return;
            } else {
                log.trace("Request not chunked");
                writeNonChunkedFile(request);
                requestComplete(event);
                return;
            }
        } else if (readingChunks){
            log.trace("Reading chunks");
            HttpChunk chunk = (HttpChunk) event.getMessage();
            if (chunk.isLast()) {
                log.trace("Read last chunk");
                readingChunks = false;
                writeChunkedFile();
                requestComplete(event);
                return;
            } else {
                log.trace("Buffering chunk content to byte buffer");
                requestContentStream.write(chunk.getContent().array());
                return;
            }
            // should not happen
        } else {
            log.error("Error handling of MessageEvent, expecting a new message or a chunk from a previous message");
        }
    } catch (Exception ex) {
        log.error("Exception: [" + ex + "]");
        sendError(context, INTERNAL_SERVER_ERROR);
    }
}

This is how I am writing the chunked requests:

private void writeChunkedFile() throws IOException {
    log.trace("Writing chunked file");
    byte[] data = requestContentStream.toByteArray();
    FileOutputStream fos = new FileOutputStream(destinationFile);
    fos.write(data);
    fos.close();
    log.debug("File upload complete, [chunked], path: [" + destinationFile.getAbsolutePath() + "] size: [" + destinationFile.length() + "] bytes");
}

This is how I send the response and close the connection:

private void requestComplete(MessageEvent event) {
    log.trace("Request complete");
    HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
    Channel channel = event.getChannel();
    ChannelFuture cf = channel.write(response);
    cf.addListener(ChannelFutureListener.CLOSE);
}

I have tried a few things in requestComplete, one being just channel.close() which didn’t seem to help. Any other thoughts or ideas?

Here is my pipeline:

@Override
public ChannelPipeline getPipeline() throws Exception {
    final ChannelPipeline pipeline = pipeline();
    pipeline.addLast("decoder", new HttpRequestDecoder());
    pipeline.addLast("encoder", new HttpResponseEncoder());
    pipeline.addLast("deflater", new HttpContentCompressor());
    pipeline.addLast("ExecutionHandler", executionHandler);

pipeline.addLast(“handler”, new FileUploadHandler());
return pipeline;
}

Thanks for any thoughts or ideas

Edit: sample log entry when logging between deflator and handler in pipeline:

2012-03-23T07:46:40.993 [New I/O server worker #1-6] WARN  NbEvents [c.c.c.r.d.l.s.h.SbApiMessageLogger.writeRequested] [] - Sending [DefaultHttpResponse(chunked: false)
HTTP/1.1 100 Continue]
2012-03-23T07:46:40.995 [New I/O server worker #1-6] WARN  NbEvents [c.c.c.r.d.l.s.h.SbApiMessageLogger.writeRequested] [] - Sending [DefaultHttpResponse(chunked: false)
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=UTF-8]
2012-03-23T07:46:41.000 [New I/O server worker #1-7] DEBUG NbEvents [c.c.c.r.d.l.s.h.SbApiMessageLogger.messageReceived] [] - Received [PUT /a/deeper/path/testFile.txt HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: 192.168.0.1:8080
Accept: */*
Content-Length: 256000
Expect: 100-continue

Answer 1

This ended up being a problem with my implementation, not related to any of the code posted here, the logic posted here seems sound and works fine. That said, many thanks to all for the helpful comments!