I use stringstream and libcurl to download data. I have a function for parsing too.
bool parse()
{
istringstream temp(buff.str());
buff.str("");
string line;
QString line_QStr, lyrics_QStr;
while (temp.good())
{
getline(temp, line);
if (QString::fromStdString(line).contains(startMarker)) break;
}
if (!temp.good()) return false; // something went wrong
while (temp.good())
{
getline(temp, line);
if ((line_QStr = QString::fromStdString(line)).contains(endMarker))
{
lyrics_QStr += line_QStr.remove(endMarker); // remove the </div>
break;
}
else
{
lyrics_QStr += line_QStr;
}
}
if (!temp.good()) return false;
QTextDocument lyricsHtml;
lyricsHtml.setHtml(lyrics_QStr);
lyrics_qstr = lyricsHtml.toPlainText();
return true;
}
When the text is ascii-only is ok. But if it’s unicode, then I’m losing the unicode chars somewhere in this function. And it comes out something like this:
I use string and getline instead of QTextStream and QString, as I couldn’t find any counterpart of good() function so I couldn’t make any decent error handling.
What am I doing wrong in this function that the unicode chars are lost and are displayed as 2 other chars? How can I fix it?
Thanks in advance!
EDIT: I changed the parse function to this:
bool LyricsManiaDownloader::parse()
{
wistringstream temp(string2wstring(buff.str()));
buff.str("");
wstring line;
QString line_QStr, lyrics_QStr;
while (temp.good())
{
getline(temp, line);
if (QString::fromStdWString(line).contains(startMarker)) break;
}
if (!temp.good()) return false; // something went wrong
while (temp.good())
{
getline(temp, line);
if ((line_QStr = QString::fromStdWString(line)).contains(endMarker))
{
lyrics_QStr += line_QStr.remove(endMarker); // remove the </div>
break;
}
else
{
lyrics_QStr += line_QStr;
}
}
if (!temp.good()) return false;
QTextDocument lyricsHtml;
lyricsHtml.setHtml(lyrics_QStr);
lyrics_qstr = lyricsHtml.toPlainText();
return true;
}
And the string2wstring function is
wstring string2wstring(const string &str)
{
wstring wstr(str.length(), L' ');
copy(str.begin(), str.end(), wstr.begin());
return wstr;
}
And there’s still some problem with encoding.
EDIT2: I use this function for saving data into a stringstream
size_t write_data_to_var(char *ptr, size_t size, size_t nmemb, void *userdata)
{
ostringstream * stream = (ostringstream*) userdata;
size_t count = size * nmemb;
stream->write(ptr, count);
return count;
}
I pass the std::ostringstream buff to curl, and the web page data is saved here.
Then I use a wistringstream, convert buff.str() to wstring and use it as a source for wistringstream.
The conversion from std::string to std::wstring is the decoding, isn’t it?
The Web server returns a stream of bytes alongside a header that indicates what encoding those bytes should be understood as. If you call QString::fromStdString without minding that encoding, then Qt will use Latin1 by default. In your case, the server sends UTF-8 data and parsing it as Latin1 results in the sort of broken text you gave as an example.
As a quick workaround, you can use QTextCodec::setCodecForCStrings to set the correct encoding globally. This is not threadsafe, though.
Ideally, you would decode the stream of bytes returned by the Web server BEFORE trying to parse it, and then turn it into a QString with fromStdWString. As a rule of thumb, you want to decode textual data as early as possible. See the famous article by Joel Spolsky about how to handle Unicode: http://www.joelonsoftware.com/articles/Unicode.html
EDIT: In essence, you are missing a step in your code: that of taking the stream of bytes returned by the server, and converting it into proper, non-ambiguous text.
You will probably find it useful to think about text and byte streams as completely different animals. The core difference is that text is unambiguous: it’s a clearly defined string of characters and character marks (diacritics) that exists in an immanent way, unbound to the details of your implementation. Byte streams, however, might mean anything depending on how you interpret them.
Take the bytes 0xC2 0xA3. They might mean ‘the character  followed by the character Ł’. That’s a perfectly valid interpretation. But they might also mean ‘the character £’. That’s another perfectly valid interpretation.
Those interpretations are what we call encodings. In the first case, the encoding was Windows-1250, and in the second case, UTF-8. Allow me to reiterate here that both encodings are potentially correct. Maybe the person sending you these bytes really meant to say ÂŁ. Maybe it was really £. Maybe it was even something else entirely, and without knowing the encoding, you can’t tell what that is.
The idea here is: a byte stream whose encoding you don’t know is basically useless.
Unfortunately, a lot of languages still allow you to pass byte streams around and pretend they’re text. C++ is not immune: the std::string type, despite its misleading name, really is a byte stream. Don’t let the name deceive you.
When you pass bytes around like they’re text, eventually the subsystem responsible for displaying that text will decode the bytes. (That’s an important rule of thumb: if text is being displayed, then bytes are being decoded somewhere.) Only said subsystem will generally use a default encoding (ASCII, Latin1), and if that’s not correct, well, that’s how you end up with unexpected characters.
And the core of your problem here is exactly that: you’re taking the byte stream sent to you by the Web server, discarding the encoding information that came with it, and passing the bytes blindly to Qt.
When you try to build a QString from an std::string, Qt tries to be helpful and assume a common encoding that often works. IMHO that’s not a good idea, because it leads to exactly the problem you’re having; I think it would have been better in the long run if QString required an explicit encoding.
So until then, you’ll have to fix your problem differently.
Thankfully, there’s a known correct way to go about that entire class of issues.
Remember what I said about a byte stream being meaningless without its encoding? Well, the Web server will generally send you an encoding, as part of the Content-Type HTTP header. Something like
Content-Type: text/html; charset=iso-8859-1.The charset is your encoding: here, it’s iso-8859-1, which is another name for Latin1.
(Note: if the content is HTML, the encoding may also be given in the
http-equivmeta header tag. If that tag disagrees with the HTTP header, then the HTTP header is assumed to be the correct one.)You want to use that encoding right away to convert those bytes into ‘actual’ text.
In a growing number of languages, ‘actual’ text is a specific type, distinct from byte streams. In C++, however, you’re on your own.
The standard way to manage your text is to transcode it from its initial encoding to UTF-16, and store the result in a std::wstring. The reason is that UTF-16 can store just about any text without ambiguity. (If you use UTF-32 instead, you’ll be able to store any text whatsoever, including text using rare old asian characters, at a cost of double the memory.)
Honestly, I was kind of hoping libcurl could do it for you; others libs in other languages do return properly decoded text out of the box, not bytes. But as far as I can tell, no such luck here.
But! You’re not using raw C++, you’re using Qt, and Qt comes with the tools for proper text handling.
So you’re going to convert your bytes into a QString as early as possible, while you still have the encoding at hand, and then you’ll be fine. QStrings are proper text, not byte streams; the Qt type for byte streams is QByteArray.
So, tell you what, let’s forego wstrings entirely, and just use QStrings.
In order to fix your problem — and any encoding problem you’ll ever get — you must thus:
1/ Figure out the expected encoding; in your case, you’ll parse the Content-Type header to figure out the encoding. Or maybe libcurl can give you that information itself, I don’t know.
2/ Use it right away to decode the content. In your case, decode it into a QString using a QTextCodec. Check the QTextCodec documentation for the details.
And you’re done.
stringnow contains proper, non-ambiguous text.This is long enough already, so I’ll stop there without going into the additional subtleties (what to do if the server lies about the Content-Type, what to do if the Web designer got the http-equiv tag wrong). The above approach will already solve 95% of the encoding problems you’ll ever encounter, and incidentally, put you ahead of 95% of coders out there.
Hope this helps!