In our prod server, every so often the requests to an external service start timing out and failing. This is preceded by a short period of high response time. The interesting thing is we have an array of identical servers calling the same service (Apple Purchase and/or Facebook) and only one of the nodes suffers the long response times. Hence its not just the service being slow. This problem is rare and hard to reproduce but still very concerning.
Thread count and memory count on the node looks fine and there no exceptions just time outs. Restarting the Java process will invariably fix this issue.
It seems from looking at our response time graphs that if the external service is actually slow for a while but then recovers some of our nodes still see that node as being down. This is just a guess for now as the slow response times could just be a function of a bug on our side.
I have added a code excerpt below, we create a new exchange object for every request so I am fairly sure there is no bad data caching.
public void doSomething() {
HttpExchange httpExchange = new HttpExchange();
httpExchange.setURL("SOME_URL");
httpExchange.setMethod("GET");
httpExchange.setEventListener(listener);
httpExchange.setTimeout(this.timeout);
this.client.send(httpExchange);
}
Has anyone seen a similar issue with the Jetty HttpClient?
Are there any known edge cases which can cause Jetty Http Client to behave this way?
Is there any shared object in the Jetty internals which I should look at to see if it is overloaded buggy?
FYI, I launched a Bug Tracking on the site and we finally have a good set of steps for reproducibility, a work around and a fix in the works.
However for a fix in the mean time “I am currently working around this by calling HttpExchange.cancel() in my .onExpire() event handler. Can you test this same fix?” –Wade Simmons see Bug Tracking