Question

I am trying to send a Google Protocol Buffer serialized string across an HTTP connection and receive it back ( unmodified ) where I will deserialize it. My problem seems to be with the ‘serializeToString’ method which takes my string and seems to add newline characters ( and maybe other whitespace ) to the serialized string. In the example below, I am taking the string “camera_stuff” and after serializing it I get a QString with newlines at the front. I have tried other strings with the same result only with different whitespace and newlines added. This causes problems for my deserializing operation as the whitespace is not captured in the HTTP request and so the response containing the serialized string from the server cannot be successfully decoded. I can partially decode it if I guess the whitespace in the serialized string. How can I solve this? Please see the following code – thanks.

I have a protocol buffer .proto file that looks like:

message myInfo {
  required string data = 1;
  required int32 number = 2;
}

After running the protoc compiler, I construct in it Qt like this:

// Now Construct our Protocol Buffer Data to Send
myInfo testData;
testData.set_data("camera_stuff");
testData.set_number(123456789);

I serialize my data to a string like this:

// Serialize the protocol buffer to a string
std::string serializedProtocolData; // Create a standard string to serialize the protocol buffer contents to
myInfo.SerializeToString(&serializedProtocolData); // Serialize the contents to the string
QString serializedProtocolDataAsQString = QString::fromStdString(serializedProtocolData);

And then I print it out like this:

// Print what we are sending
qDebug() << "Sending Serialized String: " << serializedProtocolDataAsQString;
qDebug() << "Sending Serialized String (ASCII): " << serializedProtocolDataAsQString.toAscii();
qDebug() << "Sending Serialized String (UTF8): " << serializedProtocolDataAsQString.toUtf8();
qDebug() << "Sending Serialized Protocol Buffer";
qDebug() << "Data Number: " << QString::fromStdString(myInfo.data());
qDebug() << "Number: " << (int)myInfo.number();

When I send my data as part of an HTTP multipart message I see those print statements like this ( notice the newlines in the printouts! ):

Composing multipart message...
Sending Serialized String:  "

camera_stuffï:" 
Sending Serialized String (ASCII):  "

camera_stuffï:" 
Sending Serialized String (UTF8):  "

camera_stuffÂÂï:" 
Sending Serialized Protocol Buffer 
Data:  "camera_stuff" 
Number:  123456789 
Length of Protocol Buffer serialized message:  22 
Loading complete...

The client deserializes the message like this:

// Now deserialize the protocol buffer
string = "\n\n" + string; // Notice that if I don't add the newlines I get nothing!
myInfo protocolBuffer;
protocolBuffer.ParseFromString(string.toStdString().c_str());
std::cout << "DATA: " << protocolBuffer.model() << std::endl;
std::cout << "NUMBER: " << protocolBuffer.serial() << std::endl;
qDebug() << "Received Serialized String: " << string;
qDebug() << "Received Deserialized Protocol Buffer";
qDebug() << "Data: " << QString::fromStdString(protocolBuffer.data());
qDebug() << "Number: " << (int)protocolBuffer.number();

The server gives it back without doing anything to the serialized string and the client prints the following:

RESPONSE:  "camera_stuffï:"  
DATA: camera_stu
NUMBER: 0
Received Serialized String:  "

camera_stuffï:" 
Received Deserialized Protocol Buffer 
Number:  "camera_stu"

Number:  0

So you see the issue is that I cannot guess the whitespace so I cannot seem to reliably deserialize my string. Any thoughts?

Answer 1

A serialized protobuf cannot be treated as a C string because it probably has embedded NULs in it. It’s a binary protocol which uses every possible octet value and can only be sent over an 8-bit clean connection. It’s also not a valid UTF-8 sequence, and cannot be serialized and deserialized as Unicode. So QString is also not a valid way of storing a serialized protobuf, and I suspect that might be causing you problems as well.

You can use std::string and QByteArray. I strongly suggest you avoid anything else. In particular, this is wrong:

protocolBuffer.ParseFromString(string.toStdString().c_str());

because it will truncate the protobuf at the first NUL. (Your test message doesn’t have any, but this will bite you sooner or later.)

As for sending the message over HTTP, you need to be able to ensure that all bytes in the message are sent as-is, which also means that you need to send the length explicitly. You didn’t include the code which actually transmits and receives the message, so I can’t comment on it (and I don’t know the Qt HTTP library well enough in any event), but the fact that 0x0A are being deleted from the front of the message suggests that you are missing something. Make sure you set the content-type in the message part correctly (not text, for example).