我正在尝试使用 Python 的 BeautifulSoup 库进行一些简单的网页抓取，在尝试解析大多数 YouTube 页面时遇到了 UnicodeDecodeError。
似乎 YouTube 正在提供包含无效字符的 HTML。当然，这对他们来说是个问题，但我认为 BeautifulSoup 的全部意义在于它可以处理不正确的页面并尽最大努力猜测结果。如果它只是丢弃无效字符，我会很高兴。我远非 Unicode 专家，我尝试过的各种魔法咒语对我encode没有decode任何好处。
是否有人对如何处理此错误有任何建议。我不想让我的代码特定于 YouTube，因为它需要处理大量用户指定的网页。
这是一个非常简单的代码片段，它演示了这个问题：
import urllib
from bs4 import BeautifulSoup
url='https://www.youtube.com/watch?v=W9MzrirPrCI'
text = urllib.urlopen(url).read()
soup = BeautifulSoup(text)
最后一行导致以下错误：
UnicodeDecodeError Traceback (most recent call last)
/cygdrive/d/home/ll-virtualenv/lib/python2.7/site-packages/Django-1.5.1-py2.7.egg/django/core/management/commands/shell.pyc in <module>()
----> 1 soup = BeautifulSoup(text)
/cygdrive/d/home/ll-virtualenv/lib/python2.7/site-packages/bs4/__init__.pyc in __init__(self, markup, features, builder, parse_only, from_encoding, **kwargs)
170
171 try:
--> 172 self._feed()
173 except StopParsing:
174 pass
/cygdrive/d/home/ll-virtualenv/lib/python2.7/site-packages/bs4/__init__.pyc in _feed(self)
183 self.builder.reset()
184
--> 185 self.builder.feed(self.markup)
186 # Close out any unfinished strings and close all the open tags.
187 self.endData()
/cygdrive/d/home/ll-virtualenv/lib/python2.7/site-packages/bs4/builder/_lxml.pyc in feed(self, markup)
193 def feed(self, markup):
194 self.parser.feed(markup)
--> 195 self.parser.close()
196
197 def test_fragment_to_document(self, fragment):
/usr/lib/python2.7/site-packages/lxml-3.1.0-py2.7-cygwin-1.7.17-i686.egg/lxml/etree.dll in lxml.etree._FeedParser.close (src/lxml/lxml.etree.c:88786)()
/usr/lib/python2.7/site-packages/lxml-3.1.0-py2.7-cygwin-1.7.17-i686.egg/lxml/etree.dll in lxml.etree._TargetParserContext._handleParseResult (src/lxml/lxml.etree.c:98085)()
/usr/lib/python2.7/site-packages/lxml-3.1.0-py2.7-cygwin-1.7.17-i686.egg/lxml/etree.dll in lxml.etree._TargetParserContext._handleParseResult (src/lxml/lxml.etree.c:97909)()
/usr/lib/python2.7/site-packages/lxml-3.1.0-py2.7-cygwin-1.7.17-i686.egg/lxml/etree.dll in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9071)()
/usr/lib/python2.7/site-packages/lxml-3.1.0-py2.7-cygwin-1.7.17-i686.egg/lxml/etree.dll in lxml.etree._handleSaxData (src/lxml/lxml.etree.c:94081)()
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd7 in position 22: invalid continuation byte