Question

I use the library bitarray to manage my bits conversion and to write a binary file in Python. The bitarray.to01() before writing to file is of length 4807100171. For some reason I can’t make sense of, after getting the bits fromfile (b.fromfile(file)) and then converted to a string of 0s and 1s with to01(), there is not only 0s and 1s in my string (\x00) and then, when I work with it, I get this error:

ValueError: invalid literal for int() with base 2: '0000000000000000\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

I wonder if there could be a size limit for the string coming from a file or some issues as such. If so, I haven’t found anything about it…

Edit:

Here’s a way to reproduce the problem:

import re
from bitarray import bitarray 

b = bitarray(4807100171)
b.setall(False) 

if re.match("^[\d]+$", b.to01()):
    print "there is only digits in this string."
else:
    print "there is not only digits in this string."

** Edit #2:

However, if I check my machine using platform.architecture() and sys.maxint, I get this:

In [1]: import platform, sys
In [5]: platform.architecture(), sys.maxint
Out[5]: (('64bit', ''), 9223372036854775807)

So, this is approximately 2^63. How come it truncates at 2^32?
I have 4GB of ram. I get that 2^32*1.16415e-10*8 (since I’m converting it to a string) ~= 4GB… But what about the fact that this is a 64bit machine?

Answer 1

ould not have memory on your machine to run the to01 method on a bitarray that size. The string will use one byte per digit (at least) – and you hae more than 2**32 digits. Since you are not swappign or getting out of memory errors, you may have hit some bug in bitarray —
But…step back!

Why on Earth woul you like a 4 billion digit string of “0” and “1”s? Print your self a Matrix themed racing track??

If you need to convert even a few hundred thousand digits to 0s and 1s , to look for some pattern, or whatever, you better doing it interactively, converting a few bytes at a time than wathever you are trying there.