Question

I have a few dynamic queue in my program which can add element at back or middle/remove from front during the processing and never stores the elements in memory once processing is done.However I want to log/serialize the dynamic state of the sequences including all removed elements for the introspection purpose.

e.g.

    struct element1{};
    struct element2{};
    std::deque<element1> v1;
    std::deque<element2> v2;

    if(some_cond1)
    {
       v1.push_back(element1{});
       //log v1.back()
       v1.push_back(element1{});
       //log v1.back()
    }
    if(some_cond2)
    {
       v2.push_back(element2{});
       //log v2.back()
       v2.push_back(element2{});
       //log v2.back()
    }
    if(another_cond)
    {
       v1.pop_front();//presently I am not logging removal of elements
       v2.pop_front();
    }
    if(a_third_cond)
    {
       v1.insert(some_pos,element1{});
       //log v1,some_pos
    }

I am looking at two alternative solution.

• Is there some data format (say JSON) which allows array/object to be distributed over different location and referenced through some unique name/id as I do not preserve all the elements ready in memory to write to a file and log only part of the sequence as I go?

e.g.

   { 
     "v1" : [],
     "v2" : [],
     append("v1"):
     [
        "elem1_v1", "elem2_v1"
     ],
     append("v2"):
     [
        "elem1_v2", "elem2_v2"
     ],
     insert("v1",1):
     [
        "elem3_v1"
     ]
   }

• Is it possible to write to multiple different locations to a single file so that each sequence tracks its own location and writes to that location so that end result is simple contiguous sequence description? (All i can think of something like a file backed SGI rope implementation)

In this case, the end result may look like,

   { 
     "v1" : ["elem1_v1","elem3_v1","elem2_v1"],
     "v2" : ["elem1_v2","elem2_v2"]
   }

Of course in a crude way , I can log all elements in a flat file, prefixing with some piece of information such as which sequence & where, and search them in whole file to reconstruct them back. However I am looking for a better robust & more structured alternative if available to deal with such problems.

Answer 1

Your best bet is sequential logging (with identifier) plus later reconstruction.

The issue of random access logging is two-fold:

• random writes are much slower than sequential writes on most disks
• how much space should you reserve ? what do you do when it’s exhausted ?

Most of the times, logging is not the primary goal of your application; it’s tacked on for investigation purposes. Because it’s not essential, it should be unintrisive:

• does not work ? well, no reason the application should go down
• slow as hell ? well, no reason the application should be slowed down
• …

Let an offline process do the heavy work and reconciliate the data.