Question

I have a very large array of floating point image data:

In[25]:= Dimensions[daylightImgd]
Out[25]= {18, 2002, 2989}

In[26]:= daylightImgd[[1, 1]][[1 ;; 10]]

Out[26]= {0.0122293, 0.0104803, 0.0103955, 0.0115533, 0.0118063, \
0.0120648, 0.0122957, 0.011398, 0.0117426, 0.0119997}

I can save this whole image array to disk successfully using DumpSave a la:

DumpSave["thisWorks.mx", daylightImgd]

Dumping this giant (861 meg file) takes less than 10 seconds. If I downsample these images, a la:

downsample[image_, f_] := Module[{w, h}, h = Dimensions[image][[1]];
  w = Dimensions[image][[2]];
  Table[image[[i, j]], {i, 1, h, f}, {j, 1, w, f}]]

In[26]:= daylightImgdDown = downsample[#, 4] & /@ daylightImgd;
In[27]:= Dimensions[daylightImgdDown]
Out[27]= {18, 500, 748}

In[28]:= daylightImgdDown[[1, 1]][[1 ;; 10]]

Out[28]= {0.0122293, 0.0118063, 0.0117426, 0.0119349, 0.0109443, \
0.0121632, 0.0121304, 0.00681408, 0.0101728, 0.00603242}

Then suddenly I can’t dumpsave anymore; the thing hangs and spins forever — or at least for many minutes, till I kill it, and maxes CPU:

In[31]:= DumpSave["broken.mx", daylightImgdDown];    (* Hangs forever *)

So far as I can determine, everything is as it should be: the downsampled images have the right dimensions; you can plot them via ArrayPlot and everything looks great; manually listing the first row looks fine. Everything, in short, appears the same as with the non-down-sampled images, yet on the much smaller dataset DumpSave hangs.

Help?

UPDATE: Comments on Michael’s answer

Wow. Thank you for the extremely thorough answer, which not only answered my question but taught me a bunch of peripheral stuff, too.

For your reference, the issue of packed-ness was a little trickier than just replacing my downsample[] with one of yours. Since the array I was trying to dump is an array of 18 images – a 3d array, in other words – and since I’m applying the downsampling via the Map operator, the packed-ness of the 3d array is false (according to PackedArrayQ) using either of your downsample rewrites.

However, if I take the output of those applications, and then pack the resultant 3d array, then the whole 3d array is packed, and only then can I DumpSave it. Weirdly, though, this final process of packing, while necessary for a successful DumpSave, barely seems to alter the size, as reported by ByteCount. Maybe this is easier in code:

In[42]:= downsample3[image_, f_] := 
 Module[{w, h}, h = Dimensions[image][[1]];
  w = Dimensions[image][[2]];
  Developer`ToPackedArray@Table[image[[i, j]], {i, 1, h, f}, {j, 1, w, f}]]

In[43]:= daylightImgdDown = downsample3[#, downsampleSize] & /@ daylightImgd;
In[44]:= ByteCount[daylightImgdDown]
Out[44]= 53966192

In[45]:= Developer`PackedArrayQ[daylightImgdDown]
Out[45]= False

In[46]:= dd = Developer`ToPackedArray[daylightImgdDown];
In[47]:= Developer`PackedArrayQ[dd]
Out[47]= True

In[48]:= ByteCount[dd]
Out[48]= 53963844

In[49]:= DumpSave["daylightImgdDown.mx", dd]; (* works now! *)

Again, thanks very much.

Answer 1

Without the actual data, an educated guess is that the reason the large array DumpSaves quickly is because it is a so-called “packed array”, that is, an array of machine-sized floating point numbers that has a very efficient representation in Mathematica. Your downsample function (due to the use of Table) does not return a packed array, which is much larger in memory, potentially larger than even the original array after being downsampled 4X. ByteCount might be illustrative there.

You can check for packed-array-ness with PackedArrayQ and attempt to pack an unpacked array with ToPackedArray, both found in the Developer context.

There are two solutions, if my guess is correct. One is to use ToPackedArray as shown:

downsample[image_, f_] := Module[{w, h}, h = Dimensions[image][[1]];
  w = Dimensions[image][[2]];
  Developer`ToPackedArray@Table[image[[i, j]], {i, 1, h, f}, {j, 1, w, f}]]

Even better is to simply replace your use of Table with Take, which should return a packed array in this case, and as an added bonus be a lot faster than using Table.

downsample[image_, f_] := Take[image, {1,-1,f}, {1,-1,f}]

You might also be interested in all the new image processing functionality in Mathematica 7.

Hope that helps!