1

我正在etree.HTML( data )像下面这样运行许多不同的data内容。但是,对于特定的data内容,lxml.etree.HTML将不会对其进行解析,而是进入无限循环并消耗 100% 的 CPU。

有谁确切地知道data下面这可能导致这种情况的原因?更重要的是,我怎样才能防止这种情况发生在无限数量的随机损坏上data

编辑:原来这是 lxml 2.7.8 及以下版本的错误(至少)。更新到 lxml 2.9.0,bug 消失了。

编辑:我知道这构成了一个无限循环,但这不是我得到的坏行为。它运行良好(作为无限循环),data内容健康。对于不健康data的内容,如下所示,会发生循环将停止并且 RAM 将开始填满,当它已满时,所有 CPU 进入等待状态。有关原始调试,请参阅此问题

#!/usr/bin/python
# -*- coding: utf-8 -*-
#

import sys
from lxml import etree



data = '''
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml">
<head>
<meta charset="UTF-8">
    <title>The 20 Most Despicable Things Gordon Ramsay Has Said and Done, Ranked -- Grub Street New York</title>

    <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="http://feedproxy.google.com/nymag/grubstreet" />



    <meta name="Headline" content="The 20 Most Despicable Things Gordon Ramsay Has Said and Done, Ranked" />
    <meta name="keywords" content="april bloomfield, el gordo, frank bruni, gordon ramsay, lawsuits, lists, marcus samuelsson, mario batali, shitlist, spotted pig, sued" />

    <meta name="description" content="Racism, fat-shaming, and vegetarian trickery." />

    <meta name="Byline" content="Sierra Tishgart" />
    <meta name="Type_of_Feature" content="" />
    <meta name="Issue_Date" content="March  8, 2013 12:50 PM" />
    <meta name="related_stories" content="The 20 Most Despicable Things Gordon Ramsay Has Said and Done, Ranked" />
    <meta name="document_type" content="Blog" />
    <meta name="category" content="Lists" />

    <link rel="image_src" href="http://pixel.nymag.com/imgs/daily/grub/2013/03/08/08-gorgon-ramsay.o.jpg/a_146x97.jpg" />
    <link rel="canonical" href="http://newyork.grubstreet.com/2013/03/20-despicable-things-gordon-ramsay.html" id="canonical" />

    <script>
        var canonicalUrl = "http://newyork.grubstreet.com/2013/03/20-despicable-things-gordon-ramsay.html";
    </script>



    <meta name="content.tags.primary" content=";network - Grub Street,;city - New York City,;tag - lists" />
    <meta name="content.tags" content=";tag - april bloomfield,;tag - el gordo,;tag - frank bruni,;tag - gordon ramsay,;tag - lawsuits,;tag - marcus samuelsson,;tag - mario batali,;tag - shitlist,;tag - spotted pig,;tag - sued" />
    <meta name="content.hierarchy" content="New York City:Grub Street" />
    <meta name="content.type" content="Blog" />
    <meta name="content.subtype" content="Blog Entry" />    


    <meta property="fb:app_id" content="206283005644" />
    <meta property="og:title" content="The 20 Most Despicable Things Gordon Ramsay Has Said and Done, Ranked" />
    <meta property="og:description" content="Racism, fat-shaming, and vegetarian trickery." /> 
    <meta property="og:image" content="http://pixel.nymag.com/imgs/daily/grub/2013/03/08/08-gorgon-ramsay.o.jpg/a_146x97.jpg"/>
    <meta property="og:url" content="http://newyork.grubstreet.com/2013/03/20-despicable-things-gordon-ramsay.html" />
    <meta property="og:type" content="article" />
    <meta property="og:site_name" content="Grub Street New York" />





    <meta name="viewport" content="width=1020">

<link type="text/css" rel="stylesheet" href="http://cache.nymag.com/css/screen/grubstreet/grubstreet-core.css" media="all" />
<link type="text/css" rel="stylesheet" href="http://cache.nymag.com/css/screen/section/daily/slideshow.css" media="all" />
<link type="text/css" rel="stylesheet" href="http://cache.nymag.com/css/screen/echo.css" media="all" />
<link type="text/css" rel="stylesheet" href="http://cache.nymag.com/css/screen/loginRegister.css" media="all" />
<link rel="stylesheet" href="http://cache.nymag.com/css/screen/advertising.css" media="all" />
<link rel="shortcut icon" href="http://images.nymag.com/gfx/grubst/favicon.ico" />

<style type="text/css">
#adsplashtop,#pushdown {padding:5px 5px;}
#pushdown {border-top:1px solid #737373}
</style>











    <!--[if IE 6]>
    <link rel="stylesheet" href="http://cache.nymag.com/css/screen/grubstreet/win-ie6.css" type="text/css" media="screen, projection" />
<![endif]-->

<!--[if IE 7]>
    <link rel="stylesheet" href="http://cache.nymag.com/css/screen/grubstreet/win-ie7.css" type="text/css" media="screen, projection" />
<![endif]-->




    <script type="text/javascript">
        var NYM = {};
        NYM.config = {};
        NYM.config.membership = {
            "service":"nym"
        };
        NYM.config.advertising = {
            "sitename":"nym.grubstreet"
        };

    </script>




<script type="text/javascript">
    var date = 'March 12, 2013 12:42:38';
    var currDate=new Date(date);
    var GRUBST = {};
    if (!NYM) {  
        var NYM = {};
        NYM.config = {};
        NYM.config.membership = {
            "service":"nym"
        };
        NYM.config.advertising = {
             "sitename":"nym.grubstreet"
        };
    }
</script>
<script type="text/javascript" src="http://cache.nymag.com/scripts/modernizr-1.7.min.js"></script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript" src="http://cache.nymag.com/scripts/jquery-ui-1.8.2.custom.min.js"></script>
<script type="text/javascript" src="http://cache.nymag.com/scripts/ad_manager.js"></script>
<script type="text/javascript" src="http://cache.nymag.com/js/2/global.js"></script>
<script type="text/javascript" src="http://cache.nymag.com/scripts/skinTakeover.js"></script>
<script type="text/javascript" src="http://cache.nymag.com/scripts/grubstreet-controls.js"></scr
'''







n = 0
while True:
    n += 1

    tree = etree.HTML( data )
    m = tree.xpath("//meta[@property]")

    print '-', n 
    for i in m:
        print n 
        #print (i.attrib['property'], i.attrib['content'])

对于快速版本,您可以使用:

import sys
from lxml import etree

print("%-20s: %s" % ('Python',           sys.version_info))
print("%-20s: %s" % ('lxml.etree',       etree.LXML_VERSION))
print("%-20s: %s" % ('libxml used',      etree.LIBXML_VERSION))
print("%-20s: %s" % ('libxml compiled',  etree.LIBXML_COMPILED_VERSION))
print("%-20s: %s" % ('libxslt used',     etree.LIBXSLT_VERSION))
print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))

我有:

OS                  : Ubuntu 12.10 (AWS)
Python              : sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0)
lxml.etree          : (3, 1, 0, 0)
libxml used         : (2, 7, 8)
libxml compiled     : (2, 7, 8)
libxslt used        : (1, 1, 26)
libxslt compiled    : (1, 1, 26)
4

2 回答 2

1

这是一种使用 lxml 解析部分 HTML 的方法。它似乎解决了似乎出现在 libxml(2、7、8)或更早版本中的挂起问题:

    parser = LH.HTMLParser()
    parser.feed(data)
    root = parser.close()
    m = root.xpath('//meta[@property]')

import sys
import lxml.html as LH
import lxml.etree as ET

data = '''
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="ie6"> <![endif]-->
<!--[if IE 7]>    <html class="ie7"> <![endif]-->
<!--[if IE 8]>    <html class="ie8"> <![endif]-->
<!--[if gt IE 8]><!--> <html> <!--<![endif]-->
<head profile="http://gmpg.org/xfn/11">
 <meta charset="UTF-8">
 <title>
     Erased US data shows 1 in 4 missiles in Afghan airstrikes now fired by drone: The Bureau of Investigative Journalism  </title>

 <meta name="description" content="Drone data has been wiped from the Air Force website.">

 <meta name="generator" content="Magicalia 2010" />
 <meta name="google-site-verification" content="bGFVI6kAZGjMNNiS6LGvBDWSGydwyWQI3gogCD4xP50" />

 <link href="http://cdn-images.mailchimp.com/embedcode/slim-081711.css" rel="stylesheet" type="text/css">
 <link rel="stylesheet" href="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/css/screen.css" type="text/css" media="screen, projection" />
 <link rel="stylesheet" href="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/css/print.css" type="text/css" media="print" />
 <link rel="stylesheet" href="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/style.css?3" type="text/css" media="screen, projection" />

 <!--[if IE]>
   <link rel="stylesheet" href="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/css/lib/ie.css" type="text/css" media="screen, projection" />
 <![endif]-->

 <!--[if lt IE 7]>
   <script defer type="text/javascript" src="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/pngfix.js"></script>
 <![endif]-->

 <!--[if gte IE 5.5]>
   <script language="javaScript" src="http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/dhtml.js" type="text/javaScript"></script>
 <![endif]-->

 <link rel="alternate" type="application/rss+xml" title="The Bureau of Investigative Journalism RSS Feed" href="http://www.thebureauinvestigates.com/feed/" />
 <link rel="pingback" href="http://www.thebureauinvestigates.com/xmlrpc.php" />

 <link rel="alternate" type="application/rss+xml" title="The Bureau of Investigative Journalism &raquo; Erased US data shows 1 in 4 missiles in Afghan airstrikes now fired by drone Comments Feed" href="http://www.thebureauinvestigates.com/2013/03/12/erased-us-data-shows-1-in-4-missiles-in-afghan-airstrikes-now-fired-by-drone/feed/" />
<link rel='stylesheet' id='mailchimp-css'  href='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/lib/mailchimp.dev.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='donate-css'  href='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/lib/donate.dev.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='tubepress-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/tubepress/src/main/web/css/tubepress.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='NextGEN-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/nextgen-gallery/css/nggallery.css?ver=1.0.0' type='text/css' media='screen' />
<link rel='stylesheet' id='shutter-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/nextgen-gallery/shutter/shutter-reloaded.css?ver=1.3.4' type='text/css' media='screen' />
<link rel='stylesheet' id='stbCSS-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/wp-special-textboxes/css/wp-special-textboxes.css.php?ver=4.3.72' type='text/css' media='all' />
<link rel='stylesheet' id='grid-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/big-brother/css/grid.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='reveal-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/big-brother/css/reveal.css?ver=3.5.1' type='text/css' media='all' />
<link rel='stylesheet' id='app-css'  href='http://www.thebureauinvestigates.com/wp-content/plugins/big-brother/css/app.css?ver=3.5.1' type='text/css' media='all' />
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-includes/js/jquery/jquery.js?ver=1.8.3'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/plugins/tubepress/src/main/web/js/tubepress.js?ver=3.5.1'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/jquery.cycle.js?ver=3.5.1'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/search.js?ver=3.5.1'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/nav/superfish.js?ver=3.5.1'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/nav/supersubs.js?ver=3.5.1'></script>
<script type='text/javascript' src='http://www.thebureauinvestigates.com/wp-content/themes/dxw_magicalia/js/home.js?ver=3.5.1'></sc
'''

if __name__ == '__main__':

    print("%-20s: %s" % ('Python',           sys.version_info))
    print("%-20s: %s" % ('lxml.etree',       ET.LXML_VERSION))
    print("%-20s: %s" % ('libxml used',      ET.LIBXML_VERSION))
    print("%-20s: %s" % ('libxml compiled',  ET.LIBXML_COMPILED_VERSION))
    print("%-20s: %s" % ('libxslt used',     ET.LIBXSLT_VERSION))
    print("%-20s: %s" % ('libxslt compiled', ET.LIBXSLT_COMPILED_VERSION))

    n = 0
    while True:
        n += 1
        print '-', n
        parser = LH.HTMLParser()
        parser.feed(data)
        root = parser.close()
        m = root.xpath('//meta[@property]')
        for i in m:
            print(n)

产量

% test.py
Python              : sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0)
lxml.etree          : (2, 3, 0, 0)
libxml used         : (2, 7, 8)
libxml compiled     : (2, 7, 8)
libxslt used        : (1, 1, 26)
libxslt compiled    : (1, 1, 26)
- 1
- 2
- 3
- 4
- 5
...
于 2013-03-12T16:55:39.230 回答
1

这与lxml.html- 检查:

tree = lxml.html.fromstring( data )
print tree
# <Element html at 0x1bb5530>
print tree.xpath("//meta[@property]")
# []

相反,看看这部分......你实际上有一个无限循环:

n = 0
while True:
    n += 1
    m = [] # never mind if you get results or not - looks like you don't though

    for i in m:
        print n
于 2013-03-12T16:58:00.147 回答