Question

Im doing some data cleansing on some messy data which is being imported into mysql.

The data contains ‘pseudo’ unicode chars, which are actually embedded into the strings as ‘u00e9’ etc.

So one field might be.. ‘Jalostotitlu00e1n’
I need to rip out that clumsy ‘u00e1n’ and replace it with the corresponding utf character

I can do this in either mysql, using substring and CHR maybe, but Im preprocssing the data via PHP, so I could do it there also.

I already know all about how to configure mysql and php to work with utf data. The problem is really just in the source data Im importing.

Thanks

Answer 1

There’s a way. Replace all uXXXX with their HTML representation and do an html_entity_decode()

I.e. echo html_entity_decode("Jalostotitlán");

Every UTF character in the form u1234 could be printed in HTML as ሴ. But doing a replace is quite hard, because there could be much false positives if there is no other char that identifies the beginning of an UTF sequence. A simple regex could be

preg_replace('/u([\da-fA-F]{4})/', '&#x\1;', $str)