Question

this is my script

# -*- coding: utf-8 -*-
from BeautifulSoup import BeautifulSoup
import urllib2

res = urllib2.urlopen('http://tazeh.net')
html = res.read()

soup = BeautifulSoup(''.join(html))

title = soup.findAll('title')
print title

when im am running this script in terminal i get wronged text like that

$ python test.py

[<title>ٞاŰ&OElig;گاŮ&Dagger; ؎بعŰ&OElig; ŘŞŘ­Ů&bdquo;Ű&OElig;Ů&bdquo;Ű&OElig; تازŮ&Dagger;</title>]

this title in utf-8 encoding and persian language

i’m newer in python what’s wrong?

Answer 1

if i add (like one of the comments suggested doing in less useful place):

html = html[:10000].decode("utf-8")

(the slice is because decode failed at an offset further into the page)

before:

soup = BeautifulSoup(html)

it prints:

[<title>پایگاه خبری تحلیلی تازه</title>]