Question

https://gist.github.com/1173528#comments

shows the structure of the data file …

the short version is

{ "img_ref" : {
  "$ref" : "mapimage",
  "$id" : ObjectId("4e454599f404e8d51c000002")
},
  "scale" : 128, "image" : "4e454599f404e8d51c000002", "tile_i" : 0, "tile_j" : 9, "w" : 9, "e" : 10, "n" : 0, "s" : 0,
  "heights" : [
    [
        0,
        2,
        0,
        1,
        515,
        0,
        256,
        ...], [...]
, _id: ObjectId("...") }

The stats() on this collection is:

{
"ns" : "ac2.mapimage_tile",
"count" : 18443,
"size" : 99513670744,
"avgObjSize" : 5395742.056281516,
"storageSize" : 100336473712,
"numExtents" : 74,
"nindexes" : 4,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 5832704,
"indexSizes" : {
    "_id_" : 786432,
    "img_ref_1_tile_i_1_tile_j_1" : 2236416,
    "image_1" : 1212416,
    "image_1_tile_i_1_tile_j_1_scale_1" : 1597440
},
"ok" : 1
}

Note the average object size, 5,395,742 bytes – or 5 MB! 5 MB for storing 16,384 ints seems pretty extreme!

Answer 1

See http://bsonspec.org/#/specification for how things get serialized in mongodb. Arrays are actually very space inefficient especially because we store the index number as a string key for each element. This is less of a problem for small arrays of large elements like strings or objects, but for large arrays of 32-bit ints it is very expensive.