Question

I have some code that I parse and extract some matrices from. Theses matrices are associated with certain tags.

What I would like to do is store these matrices with their associated tags for future access. So basically, I want to create a file/structure/array/something that gets updated, but that I can also call matrix operations from if I need to in the same run. Furthermore, the next time I run the program I should be able to call the arrays and tags that I’ve recoded so far. In the end, I want a database of matrices and tags.

Can anyone point me in the right direction to get started?

Some pseudocode would be much appreciated.

Answer 1

The type of structure that you now are storing your matrices in (in memory) is relevant.

For example you could store a bidimensional matrix using a structure such as,

struct {
    size_t w;
    size_t h;
    double **values;
    size_t n;
    char **tags;
}

and then write it to disk like,

fwrite(&(m->w), sizeof(size_t), 1, fp);
fwrite(&(m->h), sizeof(size_t), 1, fp);
for (y = 0; y < h; y++)
    fwrite(m->values[y], sizeof(double), m->w, fp);
fwrite(&(m->n), sizeof(size_t), 1, fp);
for (y = 0; y < m->n; y++)
{
    size_t len = strlen(m->tags[y]);
    fwrite(&len, sizeof(size_t), 1, fp);
    fwrite(m->tags[y], sizeof(char), len, fp);
}

and re-read by replacing fwrites with freads:

if (1 != fread(&(m->w), sizeof(size_t), 1, fp))
    // ERROR
if (1 != fread(&(m->h), sizeof(size_t), 1, fp))
    // ERROR
if (NULL == (m->values = malloc(sizeof(double *) * m->h)))
    // ERROR: OUT OF MEMORY
for (y = 0; y < h; y++)
{
    if (NULL == (m->values[y] = malloc(sizeof(double) * m->w)))
        // OOM
    if (m->w != fread(m->values[y], sizeof(double), m->w, fp))
        // File truncated error
}
if (1 != fread(&(m->n), sizeof(size_t), 1, fp))
    // Truncated
for (y = 0; y < m->n; y++)
{
    size_t len;
    fread(&len, sizeof(size_t), 1, fp);
    if (NULL == (m->tags[y] = malloc(len + 1)))
        // OOM
    if (len != fread(m->tags[y], sizeof(char), len, fp))
        // Truncated
    m->tags[y][len] = 0x0; // C strings must be zero-terminated
}

…but this covers only bidimensional matrices and requires each matrix to be stored into a file of its own.

Of course you might complicate the system even further and store a single “database” with a header, a number of matrices inside and so on, then store each matrix one after the other.

Probably, in the long run, you’d be better off checking whether the FITS standard Claudix suggested allows serializing matrices into strings. If they do, you will be able to store those strings into a real database (PostgreSQL, MySQL, or even SQLite3 so you don’t have to have an independent server). This will also allow you not to rewrite the whole whopper every time a tag gets changed, and have much more maintainable code, at the expense of writing two FITS-wrap functions that convert from a memory object to a SQLited field value and vice versa.

Direct use of FITS might present problems if FITS format doesn’t allow for variable tags to be associated to matrices and vectors.

If you can’t adapt FITS to write to string objects, you might still be able to use the code above to write a wrapper from matrix object to string (e.g. by base64-encoding the resulting buffer). Once you have the matrix as a string object, it’s just SQL from there:

INSERT INTO MatrixTable (matrixId, matrixEncoding) VALUES (...);
INSERT INTO MatrixTags (matrixId, tagName, tagValue) VALUES (2418, 'Taken', '2012-08-20 at noon');