Question

I am extracting a field on a webpage ad the tag html text content looks like this…

35 new

In python the extracted data looks like this…

35\xa0new

How to I deal with unicode in python to convert to a regular string?
“35 new”

what library to I use?

Thanks

Answer 1

You are getting unicode strings from the parser. You can replace certain characters if you prefer others. For example, your \xa0 is a non-breaking space, and you can replace it with a regular space:

text = text.replace(u"\xa0", u" ")

There could be many of these characters that you want to change, so it might be a long process of finding all the ones that occur in your data.