2

我正在尝试解析像这样的页面，我只是想获取标题之后的段落，我猜是介绍。

<table class="infobox vcard">我想要和之间的所有内容（包括段落标签）<table id="toc">。使用简单的 CSS 选择器来获取第一段：

div#bodyContent div#mw-content-text.mw-content-ltr p

并不总是有效，因为有时信息框表中的某些内容有一个段落。此外，介绍性段落的数量也会有所不同。如果有人有比我在这里想要的更好的方法，我也会接受。

--

请求的附加代码，尽可能缩短：

require HTTP::Request;
require LWP::UserAgent;

use LWP::Simple;
use HTML::Query 'Query';

my $pageurl = "http://en.wikipedia.org/wiki/Wayne_Rooney";
my $wikiurl = URI->new($pageurl);
my $wikirequest = HTTP::Request->new(GET => $wikiurl);
my $wikiua = LWP::UserAgent->new;
my $wikiresponse = $wikiua->request($wikirequest);
my $pagetoparse = $wikiresponse->content;

my $q2 = Query(text => $pagetoparse);
my @wikiintro = $q2->query('div#bodyContent div#mw-content-text.mw-content-ltr p')->get_elements();
my $pageintro;
if(@wikiintro) {
    if(index($wikiintro[0]->as_text(), "Appearances (Goals)") != -1){
        $pageintro = $wikiintro[1]->as_text();
    } else {
        $pageintro = $wikiintro[0]->as_text();
    }
} else {
    $pageintro = "unavailable";
}
4

2 回答 2

4

使用非标准模块的一种方法HTML::TreeBuilder。

内容script.pl：

#!/usr/bin/env perl

use warnings;
use strict;
use HTML::TreeBuilder;

my (@p);

## Read the web page.
my $root = HTML::TreeBuilder->new_from_url( shift ) or die qq|ERROR: Malformed URL\n|;

## Get the table tag with id = 'toc'.
my $table_toc = $root->look_down(
    id => 'toc'
);

## Get inmediate previous siblings <p> tags.
for my $node ( reverse $table_toc->left ) { 
    if ( $node->tag eq 'p' ) { 
        unshift @p, $node;
    }   
    else {
        last;
    }   
}

## Print the content without the HTML tags.
for my $p ( @p ) { 
    printf qq|%s\n|, $p->as_text;
}

运行它，提供 url 作为唯一参数：

perl-5.14.2 script.pl http://en.wikipedia.org/wiki/Wayne_Rooney

使用以下输出（我希望它接近您的期望）：

Wayne Mark Rooney (born 24 October 1985) is an English footballer who plays as a forward for Premier League club Manchester United and the England national team.
Rooney made his senior international debut in 2003 becoming the youngest player to represent England, until he got beaten by Theo Walcott. He is England's youngest ever goalscorer.[4] He played at UEFA Euro 2004 and scored four goals, briefly becoming the competition's youngest goalscorer. Rooney featured at the 2006 and 2010 World Cups and is widely regarded as his country's best player.[5][6][7][8][9][10] He has won the England Player of the Year award twice, in 2008 and 2009. As of October 2012, he has won 78 international caps and scored 32 goals, making him England's fifth highest goalscorer in history.[11] Along with David Beckham, Rooney is the most red carded player for England, having been sent off twice.[12]
Wide character in printf at script.pl line 25.
Aged nine, Rooney joined the youth team of Everton, for whom he made his professional debut in 2002. He spent two seasons at the Merseyside club, before moving to Manchester United for £25.6 million in the 2004 summer transfer window. The same year, Rooney acquired the nickname "Wazza".[13] Since then, with Rooney in the team, United have won the Premier League four times, the 2007–08 UEFA Champions League and two League Cups. He also holds two runner-up medals from the Champions League and has twice finished second in the Premier League. In April 2012, Rooney scored his 180th goal, making him United's fourth-highest goal-scorer of all time.[14]
Wide character in printf at script.pl line 25.
In 2009–10, Rooney was awarded the PFA Players' Player of the Year and the FWA Footballer of the Year. He has won the Premier League Player of the Month award five times, a record he shares with Steven Gerrard. He came fifth in the vote for the 2011 FIFA Ballon d'Or and was named in the FIFPro World 11 for 2011. Rooney has won the 'Goal of the Season' award by the BBC's Match of the Day poll on three occasions, with his bicycle kick against rivals Manchester City winning the 'Premier League Goal of the 20 Seasons' award.[15] Rooney is the third highest-paid footballer in the world after Lionel Messi and Cristiano Ronaldo, with an annual income of €20.7m (£18m) including sponsorship deals.[16]

编辑：为了得到结果，标签也使用printf qq|%s\n|, $p->as_HTML;而不是$p->as_text.

于 2012-12-16T23:03:11.810 回答
4

HTML::TreeBuilder可能是最好的工具。诀窍是在它提供的用于导航 HTML 树的许多方法之间学习和选择。

该程序似乎可以满足您的需求。它调用look_down查找具有给定类的表，该表紧接在您需要的输出之前。从这里调用返回HTML 层次结构中同一级别的该表之后right的所有元素。循环简单地打印这些元素中的每一个，直到遇到带有除.p

我LWP::UserAgent已经用.HTML::TreeBuildernew_from_url

use strict;
use warnings;

use LWP;
use HTML::TreeBuilder;

binmode STDOUT, ':utf8';

my $url = 'http://en.wikipedia.org/wiki/Wayne_Rooney';
my $ua = LWP::UserAgent->new;
my $resp = $ua->get($url);
die $resp->status_line unless $resp->is_success;

my $tree = HTML::TreeBuilder->new_from_content($resp->decoded_content);

my $start = $tree->look_down(_tag => 'table', class => 'infobox vcard');

for ($start->right) {
  last if $_->tag ne 'p';
  print $_->as_trimmed_text. "\n\n";
}

输出

Wayne Mark Rooney (born 24 October 1985) is an English footballer who plays as a forward for Premier League club Manchester United and the England national team.

Rooney made his senior international debut in 2003 becoming the youngest player to represent England, until he got beaten by Theo Walcott. He is England's youngest ever goalscorer.[4] He played at UEFA Euro 2004 and scored four goals, briefly becoming the competition's youngest goalscorer. Rooney featured at the 2006 and 2010 World Cups and is widely regarded as his country's best player.[5][6][7][8][9][10] He has won the England Player of the Year award twice, in 2008 and 2009. As of October 2012, he has won 78 international caps and scored 32 goals, making him England's fifth highest goalscorer in history.[11] Along with David Beckham, Rooney is the most red carded player for England, having been sent off twice.[12]

Aged nine, Rooney joined the youth team of Everton, for whom he made his professional debut in 2002. He spent two seasons at the Merseyside club, before moving to Manchester United for £25.6 million in the 2004 summer transfer window. The same year, Rooney acquired the nickname "Wazza".[13] Since then, with Rooney in the team, United have won the Premier League four times, the 2007–08 UEFA Champions League and two League Cups. He also holds two runner-up medals from the Champions League and has twice finished second in the Premier League. In April 2012, Rooney scored his 180th goal, making him United's fourth-highest goal-scorer of all time.[14]

In 2009–10, Rooney was awarded the PFA Players' Player of the Year and the FWA Footballer of the Year. He has won the Premier League Player of the Month award five times, a record he shares with Steven Gerrard. He came fifth in the vote for the 2011 FIFA Ballon d'Or and was named in the FIFPro World 11 for 2011. Rooney has won the 'Goal of the Season' award by the BBC's Match of the Day poll on three occasions, with his bicycle kick against rivals Manchester City winning the 'Premier League Goal of the 20 Seasons' award.[15] Rooney is the third highest-paid footballer in the world after Lionel Messi and Cristiano Ronaldo, with an annual income of €20.7m (£18m) including sponsorship deals.[16]
于 2012-12-17T02:51:43.867 回答