Hello dear Stackoverflowers! I have a problem with a project, regarding client -> server

Question 1

Hello dear Stackoverflowers!

I have a problem with a project, regarding client -> server communication. I want to transfer data from a C++ program to a server. For this, I chose HTTP as communication protocol, because it is easy to handle on webservers via PHP scripts of similar. The C++ program sends data (or commands) via HTTP POSTs to a server, the server generates a plain text response (Mime-Type text/plain) via PHP scripts. The generated responses are relatively short, contain a short success or failure message, and perhaps a little “payload” (all plain text).

Everything seems to work great on my development machine (local Apache server lampp). However, today I tried moving the server PHP scripts for testing purposes on a live webserver (virtual webserver services running Apache + PHP + MySQL) and, well, something stopped working…

The problem

One server-sided PHP script, is used to store data from the C++ application in a MySQL database. The data I want to store in the MySQL database is a raw json string (it is experiment data, that is processed later). The json string is formed by the C++ application. It is approximately 70 kB large (so it is large!) and sent via a POST multipart request to the webserver. The multipart request is formed via libcurl:

    foreach (const HttpKeyValuePair& kv, localServerCommand.httpKeyValuePairs) {
      if (curl_formadd(&httpPostFirst, &httpPostLast, CURLFORM_PTRNAME, kv.key.c_str(), 
                                                      CURLFORM_NAMELENGTH, (long) kv.key.size(),
                                                      CURLFORM_PTRCONTENTS, kv.value.c_str(), 
                                                      CURLFORM_CONTENTSLENGTH, (long) kv.value.size(),
                                                      CURLFORM_CONTENTTYPE, "text/plain",
                                                      CURLFORM_END) != 0) {
        cerr << "Error assembling form data" << endl;
      }
    }

    [...]

    CURL* curl = curl_easy_init();
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &receiveData);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &receiveBuffer);
    curl_easy_setopt(curl, CURLOPT_URL, url.c_str());

    if (useSimplePost) { // Only true if postString.size() < 200 byte 
      curl_easy_setopt(curl, CURLOPT_POST, 1);
      curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postString.c_str());
    } else {
      // Mutlipart
      curl_easy_setopt(curl, CURLOPT_HTTPPOST, httpPostFirst);
    }

    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curlErrorBuffer); 
    curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30);
    CURLcode errorCode = curl_easy_perform(curl);
    curl_easy_cleanup(curl);

(Just to resolve any doubts: I checked with wireshark… he is using the mutlipart post. The branch utilizing a simple post is only there, because I thought one could spare the network connection by leaving out the multipart headers for the more frequent, small requests.)

However, now the interesting part: My server-sided scripts never receives the json string. The field that should carry the json string, called ‘data’, is not part of the $_POST structure in PHP! To make things wierd, all other fields are there anyway. For testing purposes, I dumped the PHP $_GET, $_POST and $_FILES variables on the server in a log file and they look like this for the request in question:

  ------
   GET  
  ------
  array (
  )
  ------
   POST 
  ------
  array (
    'id' => '130',
    'nonceid' => '4656',
    'authentication' => 'fjOynwtBDE/g/llkQlSgrGUx0ttfJMarExF6E3jg0/QeRgzvp+Chr0XqEIzoK6Rm4/19Q6KIA/Lx32Ti1Y+cQhVdF70AS8GaI2i+0FO3Uj7WfFl4FotUzpbyLpD5/AUe0KOiGA==',
  )
  ------
   FILES 
  ------
  array (
  )

When using my local server, the ‘data’ field is part of $_POST. The ‘data’ field is the first field that is sent to the server, meaning it is written first via curl_formadd loop above, the first field in the TCP-stream as checked with wireshark, and also the first field that is in the $_POST array on my local server.

Server tests

After discovering this issue, I tested the server and tried to upload a file through WordPress using Firefox, to test if the server just rejects any large $_POST field. However, uploading seems to work (tested to upload a large PNG, larger than the json data I want to upload).

The next test was to make the ‘data’-field smaller. I tested to upload ~900 bytes of repetitions of the string

"shorter  amount of data with special characters +/=?=?$Â§+#+\'*>< "

which also worked (using multipart post).

The question

I would like the long ‘data’-field to be available as part of the $_POST variable like it is on my development machine. I do not know what is causing the issue. Can it be something with the mutipart mime type I am using (“text/plain”)? Are there any configurations limiting POST-FIELD sizes in Apache/PHP (I only know about overall POST size limits)?

I suspect this to be a an exotic server configuration problem. However, I do not know a lot about the long and (if you do not spend time on it) complicated httpd.conf.

Does anyone know what causes this problem and how to reproduce it on my local server? Or even how to resolve this issue?

Thanks in advance!

Question 2

Well, I found a work-around, but not the reason what is causing this filtering of large POST fields. I first did some additional testing regarding my missing POST field.

As Robbie suggested, I tried spoofing Firefox headers (Agent = “Mozilla/5.0”, …), but no success.

The next step was to zone in the “allowed packet size” which seems to be 65536 (closest interval I tested was 65000-66000 bytes); after that any field size exceeding this limit is gone.

After discovering this, I got frustrated, reopened the API Documentation of libcurl, and changed how data is sent to the server as part of multipart posts. Large post fields are now packed as files, rather as fields, which is the same way browsers do a file upload.

      if (kv.value.size() > 2048) {
        valueString = "";
        if (curl_formadd(&httpPostFirst, &httpPostLast, CURLFORM_PTRNAME, kv.key.c_str(), 
                                                        CURLFORM_NAMELENGTH, (long) kv.key.size(),
                                                        CURLFORM_BUFFERPTR, kv.value.c_str(), 
                                                        CURLFORM_BUFFERLENGTH, (long) kv.value.size(),
                                                        CURLFORM_FILENAME, kv.key.c_str(), 
                                                        CURLFORM_CONTENTTYPE, "text/plain",
                                                        CURLFORM_END) != 0) {
          cerr << "Error assembling form data" << endl;
        }

On the PHP side, this however means that my data does enter the script as a $_POST field, but as part of the $_FILES structure. However, the message authentication mechanism, I use for only allowing uploads by my program, requires all data in the $_POST structure, and that the sequence of the data fields is not changed. Therefore, in addition to sending my data as a file, I also add an empty field to my POST (via libcurl) with the same name as the file (“data=”).

Then in PHP, I added code to my “configuartion script”, that is always load first, which reassembles the original $_POST message, from the sent data files and the received $_POST data:

// Reassamble $_POST structure in case parts of it have been sent as file
foreach ($_POST as $pkey => $pvalue) {
  if (strlen($pvalue) == 0) {
    // Perhaps it was sent as file?
    if (isset($_FILES[$pkey])) {
      // Yes, as file! - lets read it in and put it back into $_POST
      $fileData = $_FILES[$pkey];

      if ($fileData["error"] != 0) {
        handleError("File transfer of file '" . $pkey . "' failed");
      }

      $postDataFileHandle = fopen($fileData["tmp_name"], "rb");
      if (!$postDataFileHandle) {
        handleError("Cannot read required \$_POST field that was sent as file");
      }
      $data = fread($postDataFileHandle, $fileData["size"]);
      if ($data === FALSE) {
        handleError("Error on reading required \$_POST field that was sent as file");
      }
      fclose($postDataFileHandle);

      $_POST[$pkey] = $data;
    }
  }
}

This works, but I still do not know, why/how POST-fields are filtered by Apache. Perhaps it is not done by Apache, but my hosting provider does some sort of deep packet filtering.

Still confused but it works now, thanks to everyone who tried to help!

Editorial Team · Answer 1 · 2026-06-17T04:42:24+00:00

Well, I found a work-around, but not the reason what is causing this filtering of large POST fields. I first did some additional testing regarding my missing POST field.

As Robbie suggested, I tried spoofing Firefox headers (Agent = “Mozilla/5.0”, …), but no success.

The next step was to zone in the “allowed packet size” which seems to be 65536 (closest interval I tested was 65000-66000 bytes); after that any field size exceeding this limit is gone.

After discovering this, I got frustrated, reopened the API Documentation of libcurl, and changed how data is sent to the server as part of multipart posts. Large post fields are now packed as files, rather as fields, which is the same way browsers do a file upload.

      if (kv.value.size() > 2048) {
        valueString = "";
        if (curl_formadd(&httpPostFirst, &httpPostLast, CURLFORM_PTRNAME, kv.key.c_str(), 
                                                        CURLFORM_NAMELENGTH, (long) kv.key.size(),
                                                        CURLFORM_BUFFERPTR, kv.value.c_str(), 
                                                        CURLFORM_BUFFERLENGTH, (long) kv.value.size(),
                                                        CURLFORM_FILENAME, kv.key.c_str(), 
                                                        CURLFORM_CONTENTTYPE, "text/plain",
                                                        CURLFORM_END) != 0) {
          cerr << "Error assembling form data" << endl;
        }

On the PHP side, this however means that my data does enter the script as a $_POST field, but as part of the $_FILES structure. However, the message authentication mechanism, I use for only allowing uploads by my program, requires all data in the $_POST structure, and that the sequence of the data fields is not changed. Therefore, in addition to sending my data as a file, I also add an empty field to my POST (via libcurl) with the same name as the file (“data=”).

Then in PHP, I added code to my “configuartion script”, that is always load first, which reassembles the original $_POST message, from the sent data files and the received $_POST data:

// Reassamble $_POST structure in case parts of it have been sent as file
foreach ($_POST as $pkey => $pvalue) {
  if (strlen($pvalue) == 0) {
    // Perhaps it was sent as file?
    if (isset($_FILES[$pkey])) {
      // Yes, as file! - lets read it in and put it back into $_POST
      $fileData = $_FILES[$pkey];

      if ($fileData["error"] != 0) {
        handleError("File transfer of file '" . $pkey . "' failed");
      }

      $postDataFileHandle = fopen($fileData["tmp_name"], "rb");
      if (!$postDataFileHandle) {
        handleError("Cannot read required \$_POST field that was sent as file");
      }
      $data = fread($postDataFileHandle, $fileData["size"]);
      if ($data === FALSE) {
        handleError("Error on reading required \$_POST field that was sent as file");
      }
      fclose($postDataFileHandle);

      $_POST[$pkey] = $data;
    }
  }
}

This works, but I still do not know, why/how POST-fields are filtered by Apache. Perhaps it is not done by Apache, but my hosting provider does some sort of deep packet filtering.

Still confused but it works now, thanks to everyone who tried to help!

Sign Up

Sign In

Forgot Password

The Archive Base Latest Questions

Hello dear Stackoverflowers! I have a problem with a project, regarding client -> server

The problem

Server tests

The question

Leave an answerCancel reply

1 Answer

Leave an answer
Cancel reply