Question

How do I convert extremely large (>1MB) decimal numbers to bytes/hex/binary?

For example, the number “300” should be converted to {0x01, 0x2C}. The byte order doesn’t matter and {0x2C, 0x01} is also OK.

The source numbers are stored in a prepped file (Without puntuation, whitespace or linebreaks). The largest is just over 17MB, although I can’t rule out that I’ll have a 100MB number in the future. The destination is also a file.

Is there a way that doesn’t take ages, or is fail-safe in case it does take ages?

I fear using BigInteger will take ages and is not fail-save (ie. i can’t resume half-way if something goes wrong)

I’m not against implementing my own algorithm, although I’m looking for something more efficient than ‘check if odd, divide by 2’. I’ve seen a very efficient implementation of binary to BCD, the Shift and Add-3 Algorithm, and am looking for a similarly efficient implementation in reverse.

An extra kudos for an implementation that also supports fixed-point numbers (with 1 digit and the rest decimals, eg. Pi).

Answer 1

For me BigInteger converts 1M digits into byte [] in ~39 seconds. Is it too much for you?

Random r = new Random ();

StringBuilder sb = new StringBuilder();

for (int i = 0; i < 1000000; i++)
    sb.append ("0123456789".charAt(r.nextInt(10)));

long t = System.currentTimeMillis();
BigInteger bi = new BigInteger (sb.toString());
byte [] bytes = bi.toByteArray();
System.out.println(System.currentTimeMillis() - t);

About decimals. Lets assume you have a big decimal in form <n digits>.<m digits>. You want to convert it to binary with k bits after dot. You need to solve an equation: D/(10^m) = X/(2^k), where X is integer. Here D is your decimal without dot (mantissa of your decimal), X is your binary without dot (mantissa of your binary). The equation is easy to solve: X ~ round(D*(2^k)/(10^m)). X has to be integer, so we added round().

For example you need to convert 12.34 into binary with 3 bits after dot.

n = 2
m = 2
D = 1234
k = 3
X ~ round(D*(2^k)/(10^m)) = round(1234 * 8 / 100) = round(98.72) = 99 = 1100011b

Remember that we desired 3 bits after dot, so our answer is 12.34 ~ 1100.011b

All these calculations can be done using BigInteger.