0

我试图找出用 perl分割 CLF 格式行（apache access.log 文件中的链接）的最快方法。多年来，他们已经积累了数百万。以下是我到目前为止测试过的内容。我的最后一次尝试已经比使用正则表达式更快了。

但是 - 你怎么看 -有没有办法更快地做到这一点？

1 2 3 4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 200 202
1: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
1.2.3.4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 200 202
2: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
202 200 1.2.3.4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0
3: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
1.2.3.4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 200 202
4: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
GET / ..?,-" HTTP/1.0 13/Jun/2007:03:20:15 +0200 1.2.3.4 - - 200 202
5: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 1.2.3.4 - - 200 202
6: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
---- hit <ENTER> to start Test ----
Benchmark: timing 100000 iterations of Method 1, Method 2, Method 3,
Method 4, Method 5, Method 6...
1: 39 wallclock s(37.64usr + 0.12sys = 37.77CPU) @2647.81/s(n=100000)
2: 39 wallclock s(38.35usr + 0.19sys = 38.53CPU) @2595.18/s(n=100000)
3: 39 wallclock s(37.19usr + 0.14sys = 37.33CPU) @2678.74/s(n=100000)
4: 38 wallclock s(36.80usr + 0.08sys = 36.88CPU) @2711.57/s(n=100000)
5: 38 wallclock s(36.93usr + 0.14sys = 37.07CPU) @2697.89/s(n=100000)
6: 38 wallclock s(36.11usr + 0.16sys = 36.27CPU) @2757.10/s(n=100000)

8X----------------

#!/usr/bin/perl -w
use strict;
use warnings;
use FileHandle;
use Date::Parse;
use Benchmark;

STDOUT->autoflush(1); #....................................... autoflush STDOUT

our $s='1.2.3.4 - - [13/Jun/2007:03:20:15 +0200] "GET / ..?,-" HTTP/1.0" 200 202';
our (@T,$host,$timestamp,$request);

print "---- test functionality -----------------------------------\n";

split1(); print join(" ",@T)."\n1: [$host] [$timestamp] [$request]\n";
split2(); print join(" ",@T)."\n2: [$host] [$timestamp] [$request]\n";
split3(); print join(" ",@T)."\n3: [$host] [$timestamp] [$request]\n";
split4(); print join(" ",@T)."\n4: [$host] [$timestamp] [$request]\n";
split5(); print join(" ",@T)."\n5: [$host] [$timestamp] [$request]\n";
split6(); print join(" ",@T)."\n6: [$host] [$timestamp] [$request]\n";

print "---- hit <ENTER> to start Test ----"; <>;

timethese (
  100000,
  {'1' => '&split1',
   '2' => '&split2',
   '3' => '&split3',
   '4' => '&split4',
   '5' => '&split5',
   '6' => '&split6',
  }
);

exit(0);

1;

sub split1
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  @T = $s =~ m/^(\d+)\.(\d+)\.(\d+)\.(\d+) (\S+) (\S+) \[(.+)\] "(\S+) (.*?) (\S+)" (\S+) (\S+)$/;
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",@T));
  $timestamp=str2time($T[6]);
  $request=join(" ",$T[7],$T[8],$T[9]);
}

sub split2
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  @T=split(/ /,$s); 
  splice(@T,5,@T-7,join(" ",@T[5..(@T-3)]));
  splice(@T,3,2   ,join(" ",@T[3..4     ])); 
  chomp($T[6]); $T[3]=substr($T[3],1,-1); $T[4]=substr($T[4],1,-1);
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",split(/\./,$T[0]))); 
  $timestamp=str2time($T[3]);
  $request=$T[4];
}

sub split3
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  my $i; my $x=$s; 
  $i=rindex($x,' ');push(@T,substr($x,$i+1)); $x=substr($x,0,$i);
  $i=rindex($x,' ');push(@T,substr($x,$i+1)); $x=substr($x,0,$i);
  $i=index($x,' ');push(@T,substr($x,0,$i));  $x=substr($x,$i+1,-1);
  $i=index($x,' ');push(@T,substr($x,0,$i));  $x=substr($x,$i+1);
  $i=index($x,' ');push(@T,substr($x,0,$i));  $x=substr($x,$i+2);
  $i=index($x,']');push(@T,substr($x,0,$i));  push(@T,substr($x,$i+3));
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",split(/\./,$T[2])));
  $timestamp=str2time($T[5]);
  $request=$T[6];
}

sub split4
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  my $i; my $x=$s;
  $i=rindex($x,' ');$T[6]=substr($x,$i+1); $x=substr($x,0,$i);
  $i=rindex($x,' ');$T[5]=substr($x,$i+1); $x=substr($x,0,$i);
  $i= index($x,' ');$T[0]=substr($x,0,$i); $x=substr($x,$i+1,-1);
  $i= index($x,' ');$T[1]=substr($x,0,$i); $x=substr($x,$i+1);
  $i= index($x,' ');$T[2]=substr($x,0,$i); $x=substr($x,$i+2);
  $i= index($x,']');$T[3]=substr($x,0,$i); $T[4]=substr($x,$i+3);
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",split(/\./,$T[0])));
  $timestamp=str2time($T[3]);
  $request=$T[4];
}

sub split5
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  my ($i,$j); my $x=$s;
  $i=index($x,'"')+1;
  $j=rindex($x,'"');
  $T[0]=substr($x,$i,$j-$i); 
  my $a=substr($x,0,$i-3);
  $i=rindex($a,'[');
  $T[1]=substr($a,$i+1); $a=substr($a,0,$i-1);
  $x=$a.substr($x,$j+1);
  push(@T,split(/ /,$x));      
  #----------------------------------------------------------------------------
  $request=$T[0];
  $timestamp=str2time($T[1]);
  $host=unpack("N",pack("C4",split(/\./,$T[2])));
}

sub split6
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  my ($i,$j); my $x=$s;
  $i=index($x,'[');
  $j=rindex($x,'"');
  $T[0]=substr($x,$i+1,26);
  $T[1]=substr($x,$i+30,$j-$i-30);
  push(@T,split(/ /,substr($x,0,$i-1).substr($x,$j+1)));
  #----------------------------------------------------------------------------
  $timestamp=str2time($T[0]);
  $request=$T[1];
  $host=unpack("N",pack("C4",split(/\./,$T[2])));
}

8X----------------

4

3 回答 3

1

为了建立在 amon 发现的结果的基础上，str2time 是瓶颈，我（任意）选择进行第一次拆分并使用 str2time 和Time::Piece进行测试，实际上它更快。我还没有完成配置文件以查看延迟是否仍在解析器中（或者现在正在使用 OO 模块）。

#!/usr/bin/perl

use strict;
use warnings;

use FileHandle;
use Date::Parse;
use Time::Piece;
use Benchmark;

STDOUT->autoflush(1); #....................................... autoflush STDOUT

our $s='1.2.3.4 - - [13/Jun/2007:03:20:15 +0200] "GET / ..?,-" HTTP/1.0" 200 202';
our (@T,$host,$timestamp,$request);

print "---- test functionality -----------------------------------\n";

parse(); print join(" ",@T)."\n1: [$host] [$timestamp] [$request]\n";
piece(); print join(" ",@T)."\n2: [$host] [$timestamp] [$request]\n";

print "---- hit <ENTER> to start Test ----"; <>;

timethese (
  100000,
  {
   '1' => \&parse,
   '2' => \&piece,
  }
);

exit(0);

1;

sub parse
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  @T = $s =~ m/^(\d+)\.(\d+)\.(\d+)\.(\d+) (\S+) (\S+) \[(.+)\] "(\S+) (.*?) (\S+)" (\S+) (\S+)$/;
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",@T));
  $timestamp=str2time($T[6]);
  $request=join(" ",$T[7],$T[8],$T[9]);
}

sub piece
{ $host='';$timestamp='';$request='';@T=();
  #----------------------------------------------------------------------------
  @T = $s =~ m/^(\d+)\.(\d+)\.(\d+)\.(\d+) (\S+) (\S+) \[(.+)\] "(\S+) (.*?) (\S+)" (\S+) (\S+)$/;
  #----------------------------------------------------------------------------
  $host=unpack("N",pack("C4",@T));
  $timestamp=Time::Piece->strptime($T[6], '%d/%b/%Y:%H:%M:%S %z')->epoch;
  $request=join(" ",$T[7],$T[8],$T[9]);
}

在我供电不足的上网本上，我得到：

---- test functionality -----------------------------------
1 2 3 4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 200 202
1: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
1 2 3 4 - - 13/Jun/2007:03:20:15 +0200 GET / ..?,-" HTTP/1.0 200 202
2: [16909060] [1181697615] [GET / ..?,-" HTTP/1.0]
---- hit <ENTER> to start Test ----
Benchmark: timing 100000 iterations of 1, 2...
         1: 29 wallclock secs (27.58 usr +  1.03 sys = 28.61 CPU) @ 3495.28/s (n=100000)
         2: 11 wallclock secs (11.25 usr +  0.00 sys = 11.25 CPU) @ 8888.89/s (n=100000)
于 2013-03-26T02:27:02.583 回答
1

最后我发现这种方法使用

  • 一个包，拆包替代品
  • 一个小哈希和 Time::Local 'timegm_nocheck'

它比第一次尝试快了近 4.5 倍，并且每分钟拆分约 1.000.000 条 CLF 行。通过修改 timegm 功能，它可能会更快。

#!/usr/bin/perl -w
use strict;
use warnings;
use Date::Parse;
use Time::Piece;
use Time::Local 'timegm_nocheck';
use Benchmark;

our %midx = ('Jan'=>0,'Feb'=>1,'Mar'=>2,'Apr'=>3,'May'=>4,'Jun'=>5,
             'Jul'=>6,'Aug'=>7,'Sep'=>8,'Oct'=>9,'Nov'=>10,'Dec'=>11);

our $re = qr/\A
            (\d+)\.(\d+)\.(\d+)\.(\d+)
        [ ] (\S+)
        [ ] (\S+)
        [ ] \[(\d+)\/(\S+)\/(\d+):(\d+):(\d+):(\d+) [ ] (\S+)\]
        [ ] "(\S+) [ ] (.*?) [ ] (\S+)"
        [ ] (\S+)
        [ ] (\S+)
            \z/x;

my $s='1.2.3.4 - - [13/Jun/2007:03:20:15 +0200] "GET / ..?,-" HTTP/1.0" 200 202';

print "[".join('],[',split1ST($s))."]\n";
print "[".join('],[',splitCLF($s))."]\n";

[16909060],[1181697615],[/ ..?,-"],[GET],[HTTP/1.0],[200],[202],[-],[-]

[16909060],[1181697615],[/ ..?,-"],[GET],[HTTP/1.0],[200],[202],[-],[-]

print "---- hit <ENTER> to start Test ----"; <>;

timethese (
  1000000,
  { 'split1ST' => '&split1ST($s)',
    'splitCLF' => '&splitCLF($s)',
  }
);

基准测试：对 split1ST、splitCLF 的 1000000 次迭代计时...

split1ST：338 挂钟秒（329.54 usr + 0.30 sys = 329.83 CPU）@ 3031.85/s（n=1000000）

splitCLF：76 挂钟秒（73.79 usr + 0.16 sys = 73.94 CPU）@ 13523.75/s (n=1000000)

=> splitCLF 比第一次尝试快 4.46 倍

exit(0);

1;

sub split1ST
{ @T = $s =~ m/^(\d+)\.(\d+)\.(\d+)\.(\d+) (\S+) (\S+) \[(.+)\] "(\S+) (.*?) (\S+)" (\S+) (\S+)$/;
  return ( unpack("N",pack("C4",@T)), #.............................. host-IPv4
           str2time($7), #........................................... timestamp
           $9,$8,$10,$11,$12,$5,$6)  # request,method,pro,sta,bytes,authusr,usr
}

sub splitCLF 
{ shift =~ $re;
  return ( ((((($1<<8)|$2)<<8)|$3)<<8)|$4, #......................... host-IPv4 
           Time::Local::timegm_nocheck($12,$11,$10,$7,$midx{$8},$9)-$13*36, #ts
           $15,$14,$16,$17,$18,$5,$6) #request,method,pro,sta,bytes,authusr,usr
}

1;
于 2013-03-26T19:48:49.617 回答
0

我花了一个小时摆弄正则表达式，把我的头包裹在splices 和substr恐怖，甚至一些 C 代码上。然后，我做了一件至关重要的事情：

# set the benchmark iterations down to ~ 1E4
$ perl -d:NYTProf the-script.pl
$ nytprofhtml
# open ./nytprof/index.html in browser

我分析了代码（使用Devel::NYTProf）。不足为奇：解析字符串只用了很少的时间。正则表达式应用程序split1总共花费了大约 144 毫秒。然而，日期解析在str2time. 这几乎是 1:25 的关系！

结论：

过早的优化是万恶之源。– D. 克努斯

使用一个很好的、可读的正则表达式，比如

my $split1_1_regex = qr/\A
        (\d+)\.(\d+)\.(\d+)\.(\d+)
    [ ] (\S+)
    [ ] (\S+)
    [ ] \[( [^\]]+ )\]
    [ ] "(\S+ [ ] .*? [ ] \S+)"
    [ ] (\S+)
    [ ] (\S+)
\z/x;

它的执行速度与您的 (r)index/substr 恐怖一样快，但在一定程度上是自记录的，而且肯定更容易调试。这与干净、惯用的 Perl 可能是最快的 Perl 的经验一致。

然后，您可以选择要么接受str2time缓慢，要么尝试优化它。如果您管理可证明的加速，您可能需要考虑向上游发送补丁。您还可以尝试使用其他库，或编写str2time针对您的特殊用例进行优化的自己的函数。

于 2013-03-26T01:34:41.933 回答