Question

I parse a binary point cloud file and get large arrays of points and their properties. These more often then not run out of memory when allocating towards VBOs before drawing. I need a solution that handles the memory problem but is still fast enough to feed the buffers. Since the application memory space is too small, could I somehow write a ‘file’ to the SD Card and populate the buffers piece-wise from there? I can’t parse the file more than once.

Crashing lines:

-->ByteBuffer vbb = ByteBuffer.allocateDirect(lasVertices.length * 4);
vbb.order(ByteOrder.nativeOrder());
vertexBuffer = vbb.asFloatBuffer();
vertexBuffer.put(lasVertices);
vertexBuffer.position(0);

Answer 1

Have you looked at using MappedByteBuffer? This should allow you to parse the file once and write a memory optimized binary version to sdcard and read it back in chunks, fast.

// Open the file here, store it in a field
RandomAccessFile file = null;
try {
    mFile = new RandomAccessFile("/sdcard/reallyLargeFile.dat", "r");
} catch (FileNotFoundException e) {
    e.printStackTrace();
}

Where you want to render, do this:

for (int offset = 0; offset < POINTS_SIZE; offset += BLOCK_SIZE)
if (file != null)
    try {
        MappedByteBuffer buffer = file.getChannel().map(FileChannel.MapMode.READ_ONLY, offset, BLOCK_SIZE);
        vertexBuffer.put(buffer);

        vertexBuffer.position(0);

        // Render here
    } catch (IOException e) {
        e.printStackTrace();
    }

where POINTS_SIZE = total number of points * 4 ( = size of float) and BUFFER_SIZE is a (divisible) size that doesn’t crash your application. Note that this is a single threaded solution and it might be possible to amortize your cost by using multiple threads reading smaller buffers simultaneously. Also, while this might in practice turne out to be just as efficient as a bunch of seeks and reads, it yields far more elegant and maintainable code. It is also entirely possible that memory mapped files may provide a performance boost on many machines.

Hope this helps.