Question

I’m needing to load an XML document into PHP that comes from an external source. The XML does not declare it’s encoding and contains illegal characters like &. If I try to load the XML document directly in the browser I get errors like “An invalid character was found in text content” also when loading the file in PHP I get lots of warnings like: xmlParseEntityRef: no name in Entity and Input is not proper UTF-8, indicate encoding ! Bytes: 0x9C 0x31 0x21 0x3C.

It’s clear that the XML is not well formed and contains illegal characters that should be converted to XML entities.

This is because the XML feed is made up of data supplied by lots of other users and clearly it’s not being validated or reformatted before I get it.

I’ve spoken to the supplier of the XML feed and they say they are trying to get the content providers to sort it out, but this seems silly as they should be validating the input first.

I basically need to fix the XML correcting any encoding errors and converting any illegal chars to XML entities so that the XML loads problem when using PHP’s DOMDocument functions.

My code currently looks like:

  $feedURL = '3704017_14022010_050004.xml';
  $dom = new DOMDocument();
  $dom->load($feedURL);

Example XML file showing encoding issue (click to download): feed.xml

Example XML that contains chars that have not been converted to XML entities:

<?xml version="1.0"?>
<feed>
<RECORD>
<ID>117387</ID>
<ADVERTISERNAME>Test</ADVERTISERNAME>
<AID>10544740</AID>
<NAME>This & This</NAME>
<DESCRIPTION>For one day only this is > than this.</DESCRIPTION>
</RECORD>
</feed>

Answer 1

Try using the Tidy library which can be used to clean up bad HTML and XML
http://php.net/manual/en/book.tidy.php

A pure PHP solution to fix some XML like this:

<?xml version="1.0"?>
<feed>
<RECORD>
<ID>117387</ID>
<ADVERTISERNAME>Test < texter</ADVERTISERNAME>
<AID>10544740</AID>
<NAME>This & This</NAME>
<DESCRIPTION>For one day only this is > than this.</DESCRIPTION>
</RECORD>
</feed>

Would be something like this:

  function cleanupXML($xml) {
    $xmlOut = '';
    $inTag = false;
    $xmlLen = strlen($xml);
    for($i=0; $i < $xmlLen; ++$i) {
        $char = $xml[$i];
        // $nextChar = $xml[$i+1];
        switch ($char) {
        case '<':
          if (!$inTag) {
              // Seek forward for the next tag boundry
              for($j = $i+1; $j < $xmlLen; ++$j) {
                 $nextChar = $xml[$j];
                 switch($nextChar) {
                 case '<':  // Means a < in text
                   $char = htmlentities($char);
                   break 2;
                 case '>':  // Means we are in a tag
                   $inTag = true;
                   break 2;
                 }
              }
          } else {
             $char = htmlentities($char);
          }
          break;
        case '>':
          if (!$inTag) {  // No need to seek ahead here
             $char = htmlentities($char);
          } else {
             $inTag = false;
          }
          break;
        default:
          if (!$inTag) {
             $char = htmlentities($char);
          }
          break;
        }
        $xmlOut .= $char;
    }
    return $xmlOut;
  }

Which is a simple state machine noting whether we are in a tag or not and if not then encoding the text using htmlentities.

It’s worth noting that this will be memory hungry on large files so you may want to rewrite it as a stream plugin or a pre-processor.