Question

I have a java program that needs to tar/gz a large bit of data (14 gigs) and I am using commons-compress. This will be a re-occuring job every week and is automated.

When I attempt to compress something of this size I get:

java.lang.IllegalArgumentException: 13313903445=143144405525 will not fit in octal number buffer of length 11
    at org.apache.commons.compress.archivers.tar.TarUtils.formatUnsignedOctalString(TarUtils.java:212)
    at org.apache.commons.compress.archivers.tar.TarUtils.formatLongOctalBytes(TarUtils.java:265)
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.writeEntryHeader(TarArchiveEntry.java:749)
    at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:209)
    at com.broadridge.adc.core.commons.io.ADCFileUtils.addFilesToCompression(ADCFileUtils.java:144)
    at com.broadridge.adc.core.commons.io.ADCFileUtils.addFilesToCompression(ADCFileUtils.java:154)
    at com.broadridge.adc.core.commons.io.ADCFileUtils.compressFiles(ADCFileUtils.java:125)
    at com.broadridge.adc.core.commons.io.ADCFileUtils.compressFile(ADCFileUtils.java:106)

My code works fine for smaller amounts of information (around 600 megs). I’ve looked at the source code but it’s not clean to my why the error is occurring.

Does anyone have any idea why this is happening?

Answer 1

You’re exceeding some limit defined by TarConstants.SIZELEN which is 12.

What Javadoc tells us about this:

The length of the size field in a header buffer. Includes the trailing
space or NUL.

It is used by TarArchiveEntry.writeEntryHeader(byte[] outbuf) which is in your stacktrace. Later in the stack this limit is subtracted by 1 resulting in a limit of 11 which is part of your exception message.

Later in the call stack TarUtils.formatUnsignedOctalString(final long value, byte[] buffer, final int offset, final int length) is called where the limit of 11 is exceeded.

So you seem to exceed some maximum header field size.