Question

Imaging having a stack of protocols and some c/cpp code that neatly covers
sending on each layer. Each send function uses the layer below to add
another header until the whole message is eventually placed into a
continuous global buffer on layer 0:

void SendLayer2(void * payload, unsigned int payload_length)
{
  Layer2Header header;   /* eat some stack */
  const int msg_length = sizeof(header) + payload_length;

  char msg[msg_length];  /* and have some more */
  memset(msg, 0, sizeof(msg));

  header.whatever = 42;
  memcpy(&buffer[sizeof(header)], payload, payload_length);

  SendLayer1(buffer, msg_length);
}

void SendLayer1(void * payload, unsigned int payload_length)
{
  Layer1Header header;   /* eat some stack */
  const int msg_length = sizeof(header) + payload_length;

  char msg[msg_length];  /* and have some more */
  memset(msg, 0, sizeof(msg));

  header.whatever = 42;
  memcpy(&buffer[sizeof(header)], payload, payload_length);

  SendLayer0(buffer, msg_length);
}

Now the data is moved to some global variable and actually transferred:

char globalSendBuffer[MAX_MSG_SIZE];
void SendLayer0(void * payload, unsigned int payload_length)
{
  // Some clever locking for the global buffer goes here

  memcpy(globalSendBuffer, payload, payload_length);
  SendDataViaCopper(globalSendBuffer, payload_length);
}

I’d like to reduce both the stack usage and the number of memcpy()s in this
code, so I imagine something like:

void SendLayer2(void * payload, unsigned int payload_length)
{            
  Layer2Header * header = GetHeader2Pointer();
  header->whatever = 42;

  void * buffer = GetPayload2Pointer();
  memcpy(buffer, payload, payload_length);

  ...
}

My idea would be to have something at the bottom that would calculate the proper offsets for each layers header and the offset for the actual payload by continuously subtracting from MAX_MSG_SIZE and letting the upper layer code then fill in the global buffer directly from the end / right side.

Does this sound sensible? Are there alternative, perhaps more elegant approaches?

Answer 1

You may be interested in this article: “Network Buffers and Memory Management” by Alan Cox. Basically, you have the buffer and several pointers to different interesting parts of that buffer: protocol headers, data, … Initially you reserve some space for headers by setting the data pointer to (buffer_start + max_headers_size), and each layer gets a pointer nearer to the start of the buffer.

I’m sure there must be a similar description somewhere for BSD’s mbufs.

EDIT:

David Miller (Linux networking maintainer) has this article “How SKBs work”