0

我只是在玩 urllib2 和带有 utf-8 的页面。

http://www.columbia.edu/~fdc/utf8/

仅获取前 700 个字节（顶部段）

>>> import urllib2
>>> from urllib2 import HTTPError, URLError
>>> import BaseHTTPServer
>>> opener = urllib2.OpenerDirector()
>>> opener.add_handler(urllib2.HTTPHandler())
>>> opener.add_handler(urllib2.HTTPDefaultErrorHandler())
>>> response = opener.open('http://www.columbia.edu/~fdc/utf8/')
>>> content = response.read(700)

现在从这里开始，我认为内容 var 中的字符串将是 utf-8 编码的，并且应该显示得很好。

然而

>>> content
'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">\n<html>\n<head>\n<BASE href="http://kermit.columbia.edu">\n<META http-equiv="Content-Type" content="text/html; charset=utf-8">\n<title>UTF-8 Sampler</title>\n</head>\n<body bgcolor="#ffffff" text="#000000">\n<h1><tt>UTF-8 SAMPLER</tt></h1>\n\n<big><big>&nbsp;&nbsp;\xc2\xa5&nbsp;\xc2\xb7&nbsp;\xc2\xa3&nbsp;\xc2\xb7&nbsp;\xe2\x82\xac&nbsp;\xc2\xb7&nbsp;$&nbsp;\xc2\xb7&nbsp;\xc2\xa2&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa1&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa2&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa3&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa4&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa5&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa6&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa7&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa8&nbsp;\xc2\xb7&nbsp;\xe2\x82\xa9&nbsp;\xc2\xb7&nbsp;\xe2\x82\xaa&nbsp;\xc2\xb7&nbsp;\xe2\x82\xab&nbsp;\xc2\xb7&nbsp;\xe2\x82\xad&nbsp;\xc2\xb7&nbsp;\xe2\x82\xae&nbsp;\xc2\xb7&nbsp;\xe2\x82\xaf&nbsp;\xc2\xb7&nbsp;&#8377</big></big>\n\n\n\n<p>\n<blockquote>\nFrank da Cruz<br>\n<a hre'

似乎 html 转义了，所以

>>> import HTMLParser
>>> h = HTMLParser.HTMLParser()
>>> h.unescape(content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/HTMLParser.py", line 390, in unescape
    return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

所以我不明白。我什至尝试过 .encode('utf-8') 不转义，但类似的错误。

从网站显示 utf-8 内容的最佳方式是什么？

4

2 回答 2

3

You need to decode the page from UTF-8 to Unicode; there are UTF-8 sequences in there (next to non-breaking-space HTML entities):

>>> print h.unescape(content.decode('utf8'))
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>  ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · ₯ · &#8377</big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre

You got encoding and decoding confused; the content is already UTF-8 encoded.

Note that the &#8377 is an error in the page itself, the ; was omitted. A HTML5 parser or browser would probably assume that the ; can be added and decode it anyway:

>>> print h.unescape('&#8377;')
₹

You'd have to fix those entities with a regular expression first:

>>> import re
>>> brokenrefs = re.compile(r'(&#x?[a-e0-9]+)\b', re.I)
>>> print h.unescape(brokenrefs.sub(r'\1;', content.decode('utf8')))
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>  ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · ₯ · ₹&lt;/big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre
于 2013-01-21T11:46:06.530 回答
0

你误解了你的输出。那里没有任何 HTML 编码：但是当您简单地输入contentREPL 时，它会向您显示repr()文本的 -ed 版本。

做print content给你你所期望的：

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<BASE href="http://kermit.columbia.edu">
<META http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>UTF-8 Sampler</title>
</head>
<body bgcolor="#ffffff" text="#000000">
<h1><tt>UTF-8 SAMPLER</tt></h1>

<big><big>&nbsp;&nbsp;¥&nbsp;·&nbsp;£&nbsp;·&nbsp;€&amp;nbsp;·&nbsp;$&nbsp;·&nbsp;¢&nbsp;·&nbsp;₡&amp;nbsp;·&nbsp;₢&amp;nbsp;·&nbsp;₣&amp;nbsp;·&nbsp;₤&amp;nbsp;·&nbsp;₥&amp;nbsp;·&nbsp;₦&amp;nbsp;·&nbsp;₧&amp;nbsp;·&nbsp;₨&amp;nbsp;·&nbsp;₩&amp;nbsp;·&nbsp;₪&amp;nbsp;·&nbsp;₫&amp;nbsp;·&nbsp;₭&amp;nbsp;·&nbsp;₮&amp;nbsp;·&nbsp;₯&amp;nbsp;·&nbsp;&#8377</big></big>



<p>
<blockquote>
Frank da Cruz<br>
<a hre
于 2013-01-21T11:47:24.013 回答