Question

I’m busy trying to process the following RSS feed: Yahoo Search RSS, using the following code once the data is fetched:

$response = simplexml_load_string($data);

However, 99% of the chinese characters and strings disappear when I interrogate the simple xml object.

I’ve tried converting the incoming data to utf8 by doing:

$data = iconv("UTF-8", "UTF-8//TRANSLIT", $data);

But this also doesn’t help.

Before the data hits simplexml_load_string its 100% fine. But afterwards, its not.

Any ideas?

Answer 1

What you describe sounds like an encoding issue. Encoding is like a chain, if it get’s broken at one part of the processing, the data can be damaged.

When you request the data from the RSS server, you will get the data in a specific character encoding. The first thing you should find out is the encoding of that data.

Data URL: http://tw.blog.search.yahoo.com/rss?ei=UTF-8&p=%E6%95%B8%E4%BD%8D%E6%99%82%E4%BB%A3%20%E9%9B%9C%E8%AA%8C&pvid=QAEnPXeg.ioIuO7iSzUg9wQIc1LBPk3uWh8ABnsa

According to the website headers, the encoding is UTF-8. This is the standard XML encoding.

However if the data is not UTF-8 encoded while the headers are saying so, you need to find out the correct encoding of the data and bring it into UTF-8 before you go on.

Next thing to check is if simplexml_load_string() is able to deal with UTF-8 data.

I do not use simplexml, I use DomDocument. So I can not say if or not. However I can suggest you to use DomDocument instead. It definitely supports UTF-8 for loading and all data it returns is encoded in UTF-8 as well. You should safely assume that simplexml handles UTF-8 properly as well however.

Next part of the chain is your display. You write that your data is broken. How can you say so? How do you interrogate the simplexml object?

---

Revisiting the Encoding Chain

As written, encoding is like a chain. If one element breaks, the overall result is damaged. To find out where it breaks, each element has to be checked on it’s own. The encoding you aim for is UTF-8 here.

• Input Data: All Checks OK:
• Check: Does the encoding data seems to be UTF-8? Result: Yes. The input data aquired from the data URL given, does validate the UTF-8 encoding. This could be properly tested with the data provided.
• Check: Does the raw xml data mark itself as being UTF-8 encoded? Result: Yes. This could be verified within the first bytes that are: <?xml version="1.0" encoding="UTF-8" ?>.
• Simple XML Data:
• Check: Does simple_xml support the UTF-8 encoding? Result: Yes.
• Check: Does simple_xml return values in the UTF-8 encoding? Result: Yes and No. Generally simple_xml support properties containing text that is UTF-8 encoded, however a var_dump() of the simple_xml object instance with the xml data suggests that it does not support CDATA. CDATA is used in the data in question. CDATA elements will get dropped.

At this point this looks like the error you are facing. However you can convert all CDATA elements into text. To do this, you need to specify an option when loading the XML data. The option is a constant called LIBXML_NOCDATA and it will merge CDATA as text nodes.

The following is an example code I used for the tests above and demonstrates how to use the option:

$data_url = 'http://tw.blog.search.yahoo.com/rss?ei=UTF-8&p=%E6%95%B8%E4%BD%8D%E6%99%82%E4%BB%A3%20%E9%9B%9C%E8%AA%8C&pvid=QAEnPXeg.ioIuO7iSzUg9wQIc1LBPk3uWh8ABnsa';
$xml_data = file_get_contents($data_url);

$inspect = 256;
echo "First $inspect bytes out of ", count($xml_data),":\n", wordwrap(substr($xml_data, 0, $inspect)), "\n";
echo "UTF-8 test: ", var_dump(can_be_valid_utf8_statemachine($xml_data)), "\n";

$simple_xml = simplexml_load_string($xml_data, null, LIBXML_NOCDATA);
var_dump($simple_xml);


/**
 * Bitwise check a string if it would validate 
 * as utf-8.
 *
 * @param string $str
 * @return bool
 */
function can_be_valid_utf8_statemachine( $str ) { 
    $length = strlen($str); 
    for ($i=0; $i < $length; $i++) { 
        $c = ord($str[$i]); 
        if ($c < 0x80) $n = 0; # 0bbbbbbb 
        elseif (($c & 0xE0) == 0xC0) $n=1; # 110bbbbb 
        elseif (($c & 0xF0) == 0xE0) $n=2; # 1110bbbb 
        elseif (($c & 0xF8) == 0xF0) $n=3; # 11110bbb 
        elseif (($c & 0xFC) == 0xF8) $n=4; # 111110bb 
        else return false; # Does not match 
        for ($j=0; $j<$n; $j++) { # n bytes matching 10bbbbbb follow ? 
            if ((++$i == $length) || ((ord($str[$i]) & 0xC0) != 0x80)) 
                return false; 
        } 
    } 
    return true; 
}

I assume that this will fix your issue. If not DomDocument is able to handle CDATA elements. As the encoding chain is not further tested, you might still get encoding issues in the further processing of the data, so take care that you keep the encoding up to the output.