Question

I need to scrape some data from webpages. But I have some encoding problems with it.

Here is just a litle sample code to show the problem on a well known german webpage.

I expected to get this text from the webpage:
Alle Kritiker werden gespannt nach Wolfsburg schauen, denn der VfL wurde kräftig umgekrempelt. Können die Kölner daraus ihren Nutzen ziehen?

But as you can see in my tests, I get this:
Alle Kritiker werden gespannt nach Wolfsburg schauen, denn der VfL wurde kräftig umgekrempelt. Können die Kölner daraus ihren Nutzen ziehen?

The meta tag of the page says, that it is UTF-8 encoded…
And mb_detect_encoding also says, that it is UTF-8.

But why I get this crappy text back?

And when I convert the text to ISO-8859-1 I get the expected result…

<?php
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">';

$url = "http://www.goal.com/de/match/60952/wolfsburg-vs-1-fc-k%C3%B6ln/preview";

$fileContent = @file_get_contents($url);

$dom = @DOMDocument::loadHTML($fileContent);
$xpath = new DOMXpath($dom);

$element = $xpath->query(".//*[@id='article_headline']/h2");
if ($element->length > 0) {
  $item = $element->item(0);

  $text = $item->textContent;
  echo $text . "<br>";

  $text =  iconv("UTF-8", 'ISO-8859-1', $text);
  echo $text . "<br>";
}

?>

Answer 1

DOMDocument’s html parser (which is libxml2) will try to guess the encoding of the input if it comes across malformed html. Usually it does a pretty good job but this page seems to be a pathological case. Perhaps the presence of east Asian characters is confusing it.

In situations like these where you are absolutely sure you know the encoding you can force the text into 7-bit ascii before feeding it to the loadHTML() method. You can do this like so:

$fileContent = mb_convert_encoding($fileContent, 'HTML-ENTITIES', 'UTF-8');

This will convert all non-ascii characters into an html named or numeric character entity. The page works properly for me when I do this.