Question

I am used to use JSON and Numpy to store arrays, lists and dicts in python, but I want to use BSON since floating point numbers would occupy only 4 bytes and thus reduce filesize.

With Json, I do the following:

import numpy
import json

a = numpy.random.rand(12).reshape((3,4))

with open('out.json', 'w') as out:
    json.dump(a.tolist(), out)

with open('out.json') as inp:
    b = numpy.array(json.load(inp))

print b

I didn’t find an obvious way to do the same with BSON. I tried this:

import numpy
from bson import BSON

a = numpy.random.rand(12).reshape((3,4))

b = BSON.encode({'a': a.tolist()})

with open('out.bson', 'wb') as out:
    out.write(b)

with open('out.bson', 'rb') as inp:
    print BSON().decode(inp.read())

But get this error:

Traceback (most recent call last):
  File "apaga.py", line 12, in <module>
    print BSON().decode(inp.read())
  File "/usr/lib/python2.7/dist-packages/bson/__init__.py", line 539, in decode
    (document, _) = _bson_to_dict(self, as_class, tz_aware)
bson.errors.InvalidBSON: not enough data for a BSON document

Answer 1

The version of BSON I have installed won’t import as shown so maybe I am using a different version. To see your documentation in python type help(bson) after importing…

Something like this should work:

import bson
>>> a = numpy.random.rand(12).reshape((3,4))
>>> b = bson.dumps({'a':a.tolist()})
>>> print bson.loads(b)
{u'a': [[0.033390565943162254, 0.7282666963459123, 0.03719924011978737, 0.2664821209717694], [0.6145164300761253, 0.3662769247564551, 0.5973843055182299, 0.42908933503924207], [0.05901830243140804, 0.31533731904861184, 0.7158207045507905, 0.12686922689849378]]}