Question

In a project that implements an Amazon S3 access library with the use of libcurl, I have problems with UTF8. The method for listing a bucket’s contents sends the appropriate request to the S3 server, correctly signed and all. I receive a xml document, but the data is corrupted.

I save it into a std::string.
For example, it starts with the following fragment:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult

After the last “t” of “ListBucketResult”, there is a “0” (zero) in the code, terminating the std::string. Viewing the contents of the string in the debugger or writing them into a file shows this, and many more zeros at different positions, e.g. at some (but not all) “>” closing brackets.

I use MS Visual Studio 2008 running on WinXP, the project is compiled with unicode support.

What should I do to receive proper UTF8 inside the std::string (which should be unicode agnostic, according to several sources)? Any hints on this one?

bool Http::Download(std::string& url, std::string& targetString, std::vector<std::string>* customHeaders)
{
    CURLcode result = CURLE_FAILED_INIT;
    dl = true;

    if (curl)
    {
        curl = curl_easy_init();

        curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
        curl_easy_setopt(curl, CURLOPT_HEADER, 0);
        curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteData);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &targetString);

        if (unsafe)
        {
            curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
            curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
        }

        if (customHeaders)
        {
            curl_slist* headers = 0;

            for (std::vector<std::string>::const_iterator iter = customHeaders->begin(); iter != customHeaders->end(); iter++)
            {
                headers = curl_slist_append(headers, (*iter).c_str());
                headers = curl_slist_append(headers, "\n");
            }

            curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
        }

        result = curl_easy_perform(curl);

        long http_code = 0;
        curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &http_code);
        lastHttpResult = static_cast<int>(http_code);

        curl_easy_cleanup(curl);
    }

    return (result == CURLE_OK);
};

size_t Http::WriteData(char* data, size_t size, size_t nmemb, void* target)
{
    if(target)
    {
        reinterpret_cast<std::string*>(target)->append(data);
        size_t len = size * nmemb;
        return len;
    }

    return 0;
};

Answer 1

It is quite likely that this line is part of the problem:

reinterpret_cast<std::string*>(target)->append(data);

data is not NULL terminated, so who knows what you’re putting into your string. Replace it with this:

reinterpret_cast<std::string*>(target)->append(data, size * nmemb);