Question

So I have this Java program that I use to munch through several terabytes of data. Performance is a concern.

I’ve profiled the app, and a large fraction of all memory allocations as well as a large fraction of CPU time come from performing one simple operation:

I have an array of ASCII chars. I know that the characters from offset i to offset j represent a floating-point number. I need to extract that floating-point number into a double.

The naive Double.parseDouble(new String(buf, i, j - i)) does the job. However, this is where a lot of time is spent and a lot of memory allocations come from, probably because:

• new String() creates a new object, creates an internal char[] array
  and copies the characters into the array;
• Double.parseDouble()
  creates a FloatingDecimal object, and too creates a char[] array,
  also copying the characters into it.

All these allocations and all this copying are not really necessary. Can I avoid them?

What I’d really like is a strtod-like function that would take a char[] (or a byte[]) as well as start/end offsets, and return a double.

Any suggestions? Should I roll out my own? Should I write a JNI wrapper around strtod? Should I use some Java library that’s already out there?

Answer 1

What I have done in the past is write a parser for ByteBuffer (to avoid byte to char encoding conversion) to double and visa-versa. If you can avoid creating any objects it can be much faster. This approach works for memory mapped files avoiding some copy costs as well.

The core code looks like the following. It doesn’t handle exponents, but you can add that.

@Override
public double read() throws BufferUnderflowException {
  long value = 0;
  int exp = 0;
  boolean negative = false;
  int decimalPlaces = Integer.MIN_VALUE;
  while (true) {
    byte ch = buffer.get();
    if (ch >= '0' && ch <= '9') {
      while (value >= MAX_VALUE_DIVIDE_10) {
        value >>>= 1;
        exp++;
      }
      value = value * 10 + (ch - '0');
      decimalPlaces++;
    } else if (ch == '-') {
      negative = true;
    } else if (ch == '.') {
      decimalPlaces = 0;
    } else {
      break;
    }
  }

  return asDouble(value, exp, negative, decimalPlaces);
}

The full code

It stops as soon as it gets any byte it doesn’t expect e.g. a , or \n