Question

I have 1GB binary file which basically contains 3D cube of same type of values. Saving this kind of cube with different order ([x,y,z] or [z x, y]) takes a lot of time with fseek and fwrite. But one of software packages does this a lot faster than my program. Is there any approach to make file writing faster than one with fseek/fwrite?

Answer 1

You should not use fseek in the inner loop of file io operations. For the writing functions to be fast they cache the writes. If you seek all over the place you keep blowing the cache.

Do all your transformations in memory – e.g rotate the cube in memory, and then write the file in a few sequentual fwrite calls.

If you can’t transform your data completely in memory, then assemble your cube one plane at a time in memory and write out each plane.

@edit:

In your case you don’t want to use fseek at all. Not even one.

Do something like this:

void writeCubeZYX( int* cubeXYZ, int sizeOfCubeXYZ, FILE* file )
{
   int* cubeZYX = malloc( sizeOfCubeXYZ );

   // all that monkey business you're doing with fseek is done inside this
   // function copying memory to memory. No file IO operations in here.
   transformCubeXYZ_to_ZYX( cubeXYZ, cubeZYX, sizeOfCubeXYZ );

   // one big fat very fast fwrite. Optimal use of file io cache.
   fwrite(  file, cubeZYX, 1, sizeOfCubeXYZ );

   free( cubeZYX ); // quiet pedantry.
}

@edit2:

Ok suppose you can’t transform it all in memory then transform it in planes and write out one plane at a time – in file order – that is with no fseeks.

So say an [XYZ] cube is laid out in memory as a series of Z [XY] matrices. That is the [XY] planes of your cube are contiguous in memory. And you want to write out as [ZYX]. So in the file you want to write out a series of X [ZY] matrices. Each [ZY] will be contiguous in the file.

So you do something like this:

void writeCubeZYX( int* cubeXYZ, int x, int y, int z, FILE* file )
{
   int sizeOfPlaneZY = sizeof( int ) * y * z; 
   int* planeZY = malloc( sizeOfPlaneZY );

   for ( int i = 0; i < X; i++ )
   {
      // all that monkey business you're doing with fseek is done inside this
      // function extracting one ZY plane at a time. No file IO operations in here.
      extractZYPlane_form_CubeXYZ( cubeXYZ, planeZY, i );

      // in X big fat very fast fwrites. Near optimal use of file io cache.
      fwrite(  file, planeZY, 1, sizeOfPlaneZY );
   } 

   free( planeZY ); // quiet pedantry.
}