Question

I’m trying to base64 encode a huge input file and end up with an text output file, and I’m trying to find out whether it’s possible to encode the input file bit-by-bit, or whether I need to encode the entire thing at once.

This will be done on the AS/400 (iSeries), if that makes any difference. I’m using my own base64 encoding routine (written in RPG) which works excellently, and, were it not a case of size limitations, would be fine.

Answer 1

It’s not possible bit-by-bit but 3 bytes at a time, or multiples of 3 bytes at time will do!.

In other words if you split your input file in “chunks” which size(s) is (are) multiples of 3 bytes, you can encode the chunks separately and piece together the resulting B64-encoded pieces together (in the corresponding orde, of course. Note that the last chuink needn’t be exactly a multiple of 3 bytes in size, depending on the modulo 3 value of its size its corresponding B64 value will have a few of these padding characters (typically the equal sign) but that’s ok, as thiswill be the only piece that has (and needs) such padding.

In the decoding direction, it is the same idea except that you need to split the B64-encoded data in multiples of 4 bytes. Decode them in parallel / individually as desired and re-piece the original data by appending the decoded parts together (again in the same order).

Example:

“File” contents =
"Never argue with the data." (Jimmy Neutron).
Straight encoding = Ik5ldmVyIGFyZ3VlIHdpdGggdGhlIGRhdGEuIiAoSmltbXkgTmV1dHJvbik=

Now, in chunks:
"Never argue     –>     Ik5ldmVyIGFyZ3Vl
with the         –>        IHdpdGggdGhl
data." (Jimmy Neutron) –> IGRhdGEuIiAoSmltbXkgTmV1dHJvbik=

As you see piece in that order the 3 encoded chunks amount the same as the code produced for the whole file.

Decoding is done similarly, with arbitrary chuncked sized provided they are multiples of 4 bytes. There is absolutely not need to have any kind of correspondance between the sizes used for encoding. (although standardizing to one single size for each direction (say 300 and 400) may makes things more uniform and easier to manage.