Question

I am building an application based on distributed linear algebra using Trilinos, the main issue is that memory consumption is much higher than expected.

I have built a simple test case for building an Epetra::VbrMatrix with 1.5 million doubles grouped as 5 millions blocks of 3 doubles, which should be about 115MB.

After building the matrix on 2 processors, half data each, I get a memory consumption of 500MB on each processor, which is about 7.5 times the data, it looks unreasonable to me, the matrix should just have some integer arrays for locating the nonzero blocks.

I asked on the trilinos-users mailing list, they say memory usage looks too high, but hope to have some more help here.

I tested both on my laptop with Ubuntu + gcc 4.4.5 + Trilinos 10.0 and on a cluster with PGI compiler and Trilinos 10.4.0, the result is about the same.

My test code is on gist https://gist.github.com/848310, where I also wrote memory consumption at different stage in my testing with 2 MPI processes on my laptop.

If anybody has any suggestion that would be really helpful. Also if you could even just build, run and report memory consumption it would be great.

Answer 1

answer by Alan Williams form the trilinos-users list, in short VBRmatrix is not suitable for such small blocks, as the storage overhead is bigger than the data themselves:

The VbrMatrix storage format
definitely incurs some storage
overhead, as compared to the simple
number of double-precision values
being stored.

In your program, you are storing
5,000,000 X 1 X 3 == 15 million
doubles. With 8 bytes per double, that
is 120 million bytes.

The matrix class Epetra_VbrMatrix
(which is the base class for
Epetra_FEVbrMatrix) internally stores
a Epetra_CrsGraph object, which
represents the sparsity structure.
This requires a couple of integers per
block-row, and 1 integer per
block-nonzero. (Your case has 5
million block-rows with 1
block-nonzero per row, so at least 15
million integers in total.)

Additionally, the Epetra_VbrMatrix
class stores a
Epetra_SerialDenseMatrix object for
each block-nonzero. This adds a couple
of integers, plus a bool and a
pointer, for each block-nonzero. In
your case, since your block-nonzeros
are small (1×3 doubles), this is a
substantial overhead. The VbrMatrix
format has proportionally less
overhead the bigger your
block-nonzeros are. But in your case,
with 1×3 blocks, the VbrMatrix is
indeed occupying several times more
memory than is required for the
15million doubles.