Question

I upgraded an existing application and in the new version I require OS 5 – one of the reasons was that I wanted to use ConnectionFactory for communicating via HTTP/HTTPS with our server without having to add all the URL parameters for using BES, BIS, Direct TCP, Wifi and so on.

The ConnectionFactory is now configured to choose the best way to connect to our services via preferred types.

My connection code looks like this:

ConnectionFactory connectionFactory = new ConnectionFactory();
BisBOptions bisOptions = new BisBOptions(BIS_SECRET);
connectionFactory.setTransportTypeOptions(TransportInfo.TRANSPORT_BIS_B, bisOptions);
connectionFactory.setConnectionMode(ConnectionFactory.ACCESS_READ_WRITE);
connectionFactory.setEndToEndDesired(true);
connectionFactory.setPreferredTransportTypes(new int[] { TransportInfo.TRANSPORT_BIS_B, TransportInfo.TRANSPORT_MDS,
TransportInfo.TRANSPORT_TCP_WIFI, TransportInfo.TRANSPORT_TCP_CELLULAR });
ConnectionDescriptor connectionDescriptor = connectionFactory.getConnection("https://myserver.com/serviceurl");
try {
  HttpConnection con = (HttpConnection) connectionDescriptor.getConnection();
  byte[] bytes = parameter.toString().getBytes(UTF_8);
  con.setRequestProperty(CONTENT_LENGTH, String.valueOf(bytes.length));
  os = con.openOutputStream();
  os.write(bytes);
  os.flush();
  int responseCode = con.getResponseCode();
  if (responseCode == 401) {
    throw new InvalidCredentialsException("Invalid credentials");
  } else if (responseCode != 200 && responseCode != 500) {
    EventLogger.logEvent(RTSID, ("Response code " + responseCode + " " + con
          .getResponseMessage()).getBytes(), EventLogger.ERROR);
    EventLogger.logEvent(RTSID, bytes, EventLogger.ERROR);
    throw new IOException("Invalid request");
  }
  is = con.openInputStream();
  if (is != null) {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    int c = 0;
    try {
      c = is.read();
    } catch (Exception ex) {
      c = -1;
    }
    while (c >= 0) {
      baos.write(c);
      try {
        c = is.read();
      } catch (Exception ex) {
        c = -1;
      }
    }
    String response = new String(baos.toByteArray(), UTF_8);
    try {
      JSONObject jsonObject;
      if (response.startsWith("[")) {
        jsonObject = new JSONObject();
        jsonObject.put(ARRAY, new JSONArray(response));
      } else {
        jsonObject = new JSONObject(response);
      }
      if (responseCode == 500) {
        throw new Exception(jsonObject.getString("message"));
      }
      return jsonObject;
    } catch (JSONException e) {
      EventLogger.logEvent(RTSID, ("Exception occured: " + e.toString()).getBytes(),
      EventLogger.ERROR);
    }
  }
} finally {
  if (is != null) {
    try {
      is.close();
    } catch (Exception e) {
    }
  }
  if (os != null) {
    try {
      os.close();
    } catch (Exception e) {
    }
  }
  if (con != null) {
    try {
      con.close();
    } catch (Exception e) {
    }
  }
}

My problem is that this works not as well as when I added the connection parameters to my URL manually. I get errors in the server logs looking like the clients close the connection after some kind of timeout.

Here are some log examples:

93.186.30.120 - - [28/Jun/2012:15:50:08 +0200] "POST /service/methodX HTTP/1.1" 400 145 "-" "myapp VendorID/301" 10012567 
93.186.22.118 - - [28/Jun/2012:16:30:56 +0200] "POST /service/methodY HTTP/1.1" 400 145 "-" "myapp VendorID/137" 10012435
74.82.68.35 - - [28/Jun/2012:16:53:23 +0200] "POST /service/methodZ HTTP/1.1" 400 145 "-" "myapp BlackBerry9650/6.0.0.524 VendorID/105" 10012644

The IP Adresses are from RIM Networks – so these are Connections comming from BIS

Those connections got status code 400 (Bad Request) from the server

The large numbers at the end of the line (e.g. 10012644) show the time the request was processed on the server in microseconds: 10012644 = about 10 seconds

Do the RIM servers add a connection timeout of 10 seconds? That seems fairly short!

The problem is difficult to reproduce – has anybody experienced something like that before?

Answer 1

I found the reason. Problem was caused by default configuration of Apache module mod_reqtimeout:

<IfModule reqtimeout_module>

# mod_reqtimeout limits the time waiting on the client to prevent an
# attacker from causing a denial of service by opening many connections
# but not sending requests. This file tries to give a sensible default
# configuration, but it may be necessary to tune the timeout values to
# the actual situation. Note that it is also possible to configure
# mod_reqtimeout per virtual host.


# Wait max 20 seconds for the first byte of the request line+headers
# From then, require a minimum data rate of 500 bytes/s, but don't
# wait longer than 40 seconds in total.
# Note: Lower timeouts may make sense on non-ssl virtual hosts but can
# cause problem with ssl enabled virtual hosts: This timeout includes
# the time a browser may need to fetch the CRL for the certificate. If
# the CRL server is not reachable, it may take more than 10 seconds
# until the browser gives up.
RequestReadTimeout header=20-40,minrate=500

# Wait max 10 seconds for the first byte of the request body (if any)
# From then, require a minimum data rate of 500 bytes/s
RequestReadTimeout body=10,minrate=500

</IfModule>

I guess BlackBerry clients wer hit harder because sending request body via RIM BIS infrastructure takes longer.

Set the value to 100 seconds and monitoring if clients are still affected.