python - 如何在 BeautifulSoup Python 中将内容编码为 HTML

Question

我尽我所能对页面进行编码，然后使用 BeautifulSoup。但是，当我运行时，它会显示 unicode 结果。谁能帮我如何在 BeautifulSoup 下编码

我的代码：

import httplib
import urllib
import urllib2
from BeautifulSoup import BeautifulSoup
import HTMLParser


headers={
'Host': 'digitalvita.pitt.edu',
'Connection': 'keep-alive',
'Origin': 'https://digitalvita.pitt.edu',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1',
'Content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Accept': 'text/javascript, text/html, application/xml, text/xml, */*',
'Referer': 'https://digitalvita.pitt.edu/index.php',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Cookie': 'PHPSESSID=lvetilatpgs9okgrntk1nvn595'
}

data={
'action':'search',
'xdata':'<search id="1"><context type="all" /><results><ordering>familyName</ordering><pagesize>100000</pagesize><page>1</page></results><terms><name>d</name><school>All</school></terms></search>',
'request':'search'
}

data = urllib.urlencode(data)
print data
req = urllib2.Request('https://digitalvita.pitt.edu/dispatcher.php', data, headers)
response = urllib2.urlopen(req)
the_page = response.read()

htmlCodes = [
    ['&', '&amp;'],
    ['<', '&lt;'],
    ['>', '&gt;'],
    ['"', '&quot;'],
]
htmlCodesReversed = htmlCodes[:]
htmlCodesReversed.reverse()

def htmlEncode(s, codes=htmlCodes):
    """ Returns the HTML encoded version of the given string. This is useful to
        display a plain ASCII text string on a web page."""
    for code in codes:
        s = s.replace(code[1], code[0])
    return s
s=htmlEncode(the_page,codes=htmlCodes)

h = HTMLParser.HTMLParser()
s=h.unescape(s)

s.encode("utf-8")

soup=BeautifulSoup(s,convertEntities=BeautifulSoup.HTML_ENTITIES)
print soup

简单的结果如下：

 <a href="#local" onclick="dvSearch.ToggleInterests(141432);"><span class="iToggle" id="toggle_141432">more...</span></a></span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Znati, Taieb</span><span class="email"> (<a href="mailto:znati@pitt.edu">znati@pitt.edu</a>) </span></div><div class="professionalPosition">Computer Science, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zoffer, H</span><span class="email"> (<a href="mailto:zoffer@pitt.edu">zoffer@pitt.edu</a>) </span></div><div class="professionalPosition">"KGSB-Dean, Office of", University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zorn, Kristin</span><span class="email"> (<a href="mailto:kzorn@mail.magee.edu">kzorn@mail.magee.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zou, Chunbin</span><span class="email"> (<a href="mailto:chz4@pitt.edu">chz4@pitt.edu</a>) </span></div><div class="researchInterest"><b>Research Interests: </b>fatty liver disease; tyrosine kinase receptor; proteasome endopeptidase complex; phosphatidylcholines; trypanosome; Fas; ubiquitin; pulmonary surfactants; HGF/Met</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zou, Xiuying</span><span class="email"> (<a href="mailto:xiz42@pitt.edu">xiz42@pitt.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zrust, Marilyn</span><span class="email"> (<a href="mailto:zrustm@pitt.edu">zrustm@pitt.edu</a>) </span></div><div class="professionalPosition">Clinical Instructor, Acute/Tertiary Care, University of Pittsburgh School of Nursing</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zubieta, Juan</span><span class="email"> (<a href="mailto:zubietajc@upmc.edu">zubietajc@upmc.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuccoli, Giulio</span><span class="email"> (<a href="mailto:giz3@pitt.edu">giz3@pitt.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckerman, Daniel</span><span class="email"> (<a href="mailto:ddmmzz@pitt.edu">ddmmzz@pitt.edu</a>) </span></div><div class="professionalPosition">Computational Biology, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>structural biology; stochastic processes; computer simulation; coarse-grained models; protein dynamics and fluctuations; models, theoretical</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckoff, Allan</span><span class="email"> (<a href="mailto:zuckoffa@pitt.edu">zuckoffa@pitt.edu</a>) </span></div><div class="professionalPosition">Psychology, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckoff, Allan</span><span class="email"> (<a href="mailto:ZuckoffAM@UPMC.EDU">ZuckoffAM@UPMC.EDU</a>) </span></div><div class="professionalPosition">Psychiatry, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>psychotherapy; substance-related disorders; motivational interviewing; grief treatment ; diagnosis, dual (psychiatry); treatment adherence; patient compliance; traumatic grief and substance abuse</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zukor, Tevya</span><span class="email"> (<a href="mailto:tez5@pitt.edu">tez5@pitt.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuley, Margarita</span><span class="email"> (<a href="mailto:zuleyml@upmc.edu">zuleyml@upmc.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zunino, Paolo</span><span class="email"> (<a href="mailto:paz13@pitt.edu">paz13@pitt.edu</a>) </span></div><div class="professionalPosition">Mech Eng and Materials Sci, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zureikat, Amer</span><span class="email"> (<a href="mailto:zureikatah@upmc.edu">zureikatah@upmc.edu</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zutter, Chad</span><span class="email"> (<a href="mailto:czutter@pitt.edu">czutter@pitt.edu</a>) </span></div><div class="professionalPosition">KGSB-Business Admin, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zyczynski, Halina</span><span class="email"> (<a href="mailto:hzyczynski@mail.magee.edu">hzyczynski@mail.magee.edu</a>) </span></div><div class="professionalPosition">Obstetrics, Gynecology and Reproductive Sciences, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>pelvic floor reconstruction; rectocele; uterine prolapse; sacralcolpopexy; bladder diseases; colpocleisis; pelic organ prolapse; urinary incontinence</div></td></tr></table></div>

]]>

Answer 1

看起来问题在于您正在混淆字符集。

我要做的第一件事就是改变你的Accept-Charset，所以你只接受 utf-8。

'Accept-Charset': 'utf-8;q=0.7,*;q=0.3',

接下来，结果response.read()是一个 8 位字符串，您必须对其进行解码。由于我们现在知道它是 utf-8，因此您可以这样做：

the_page = response.read().decode('utf-8')

通过这两个更改，当我运行您的脚本时，相同的片段返回为：

 … Self Care</span>
                                            <a href="#local" onclick="dvSearch.ToggleInterests(…

不再有垃圾 Unicode 字符。

当然，这只是因为服务器愿意返回 utf-8 才有效。对于更一般的情况，您有一些只能执行 utf-8 的服务器和其他只能执行 Latin-1 的服务器，您需要做一些更复杂的事情。不理会标Accept-Charset头，然后更改读取以查看响应标头。像这样的东西：

response = urllib2.urlopen(req)
charset = response.info().getencoding()
the_page = response.read().decode(charset)

有许多配置不当的服务器实际上不会返回字符集，即使它们没有返回纯 7 位 ASCII。在这种情况下，您需要检查服务器返回的内容并硬编码正确的答案，或者编写代码以尝试动态检测正确的字符集。希望你永远不会遇到这种情况……</p>