2

我在使用 Perl 程序时遇到了问题,该程序可用于重新组织对应用程序的访问跟踪。

我已经使用跳转行功能实现了以下解决方案,因为未来我可以拥有 10 个或更多旋转文件,每个文件 50MB。

我想跳过之前处理中已经读取的行(如果文件的 inode 没有更改),这样我将只使用增量。

我希望这段代码可以帮助其他用户。

#!/usr/bin/perl

use strict;
use warnings 'all';

use File::Path qw<mkpath>;
use File::Spec;
use File::Copy;
use POSIX qw<strftime>;
use English;

# Dynamic Variables
my %older_count;
my %older_inode;
my @newer_filelist;
my @events;

my $OLD_IN_FILE = "";

# Static Variables
# Directories
my $IN_DIR               = "/tmp/appo/log";    # Input Directories
my $OUTPUT_LOG_DIRECTORY = "/tmp/appo/A14";    # Output directory

# Files
my $SPLITTED_OUTFILE = "parse_log.csv";           # Splitted by month output file
my $R_STATS          = ".rotation_statistics";    # Rotation Statistic file

## MAIN

# Loading old statistics
if (-e $R_STATS) { 
   open (STAT_FILE, $R_STATS) or die $!;

    while ( <STAT_FILE> ) {
       my @lines = split /\n/;
       my ( $file, $inode, $nrows ) = $lines[0] =~ /\A(.\w.*);(\d.*);(\d.*)/;    # Encapsulate values

       push @{ $older_count{$file} }, $nrows;
       push @{ $older_inode{$file} }, $inode;
   }

   close( STAT_FILE );
}

# Loading new events from log
foreach my $INPUT ( glob( "$IN_DIR/logrotate_*.log" ) ) {

    my $inode        = ( stat( $INPUT ) )[1];
    my $currentinode = $older_inode{$INPUT}[0];

    my $jumprow = 0;
    $jumprow = $older_count{$INPUT}[0] if $currentinode == $inode; 

# Get current file stastistics
   if ( $INPUT ne $OLD_IN_FILE ) {
       my $count = ( split /\s+/, `wc -l $INPUT` )[0];
       push @newer_filelist, {
             filename => $INPUT,
             inode    => $inode,
             count    => $count
       };
    }

    # Log opening
    open my $fh, '<', $INPUT or die "can't read open '$INPUT': $OS_ERROR";

    $/ = "\n\n";    # record separator

    while ( <$fh> ) {

        # next unless $. > $jumprow; # This instruction doesn't work

        # Log processing
        my @lines = split /\n/;
        my $i     = 0;

        foreach my $lines ( @lines ) {

            # Take only Authentication rows and skip others
            if ( $lines[$i] =~ m/\A#\d.\d.+#\d{4}\s\d{2}\s\d{2}\s\d{2}:\d{2}:\d{2}:\d{3}#\+\d+#\w+#\/\w+\/\w+\/Authentication/ ) {

                # Shows only LOGIN/LOGOUT access type and exclude GUEST users
                if ( $lines[ $i + 2 ] =~ m/Login/ || $lines[ $i + 2 ] =~ m/Logout/ && $lines[ $i + 3 ] !~ m/Guest/ ) {

                    my ( $y, $m, $d, $time ) = $lines[$i] =~ /\A#\d.\d.+#(\d{4})\s(\d{2})\s(\d{2})\s(\d{2}:\d{2}:\d{2}:\d{3})/;

                    my ( $action ) = $lines[ $i + 2 ] =~ /(\w+)/;
                    my ( $user )   = $lines[ $i + 3 ] =~ /\w+:\s(.+)/;

                    push @events, {
                        date   => "$y/$m/$d",
                        time   => $time,
                        action => $action,
                        user   => $user
                    };  # Array loader
                }
            }
            else {
                next;
            }

            $i++;
        }

        $OLD_IN_FILE = $INPUT;
    }
    close( $fh );
}

# Print Log statistics for futher elaborations
open( STAT_FILE, '>', $R_STATS ) or die $!;

foreach my $my_filelist ( @newer_filelist ) {
    print STAT_FILE join ';', $my_filelist->{filename}, $my_filelist->{inode}, "$my_filelist->{count}\n";
}

close( STAT_FILE );

my @by_user = sort { $a->{user} cmp $b->{user} } @events;    # Sorting by users

foreach my $my_list ( @by_user ) {

    my ( $y, $m ) = $my_list->{date} =~ /(\d{4})\/(\d{2})/;

    # Generate Directory YYYY-Month - #2009-January
    my $directory = File::Spec->catfile( $OUTPUT_LOG_DIRECTORY, "$m-$y" );

    unless ( -e $directory ) {
        mkpath( $directory, { verbose => 1 } );
    }

    my $log_file_path = File::Spec->catfile( $directory, $SPLITTED_OUTFILE );

    open( OUTPUT, '>>', $log_file_path ) or die $!;
    print OUTPUT join ';', $my_list->{date}, $my_list->{time}, $my_list->{action}, "$my_list->{user}\n";
}

close( OUTPUT );

我的日志文件是

logrotate_1.0.log

#2.0^H#2018 05 29 10:09:45:969#+0200#Info#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103EC9E50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER1#5##C47731E44D00000bae##0#Thread[HTTP Worker [@1473726842],5,Dedicated_Application_Thread]#Plain##
Login
User: USER4
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 29 11:51:06:541#+0200#Info#/Sy/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103EC9F50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER4#6##A40B81404D03c0bae##0#Thread[HTTP Worker [@1264376989],5,Dedicated_Application_Thread]#Plain##
Login
User: USER1
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 30 11:54:03:906#+0200#Info#/Sy/Sec/Informtion#
#BC-JAS-SEC#security#C0000A7103EC9F50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER4#6##A40B81404D03c0bae##0#Thread[HTTP Worker [@1264376989],5,Dedicated_Application_Thread]#Plain##
Login
User: USER4
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 30 11:59:59:156#+0200#Info#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA0C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER3#7##9ACF7Ec0bae##0#Thread[HTTP Worker [@124054179],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER3
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 30 08:32:11:348#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20E0000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#03c0bae##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Login
User: USER4
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 30 11:09:54:978#+0200#Info#/Sys/Sec/Information#
#BC-JAS-SEC#security#C0000A7103ECA20E0000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#03c0bae##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Login
User: USER2
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 06 01 08:11:30:008#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER2
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 06 01 11:11:29:658#+0200#Info#/Sys/Sec/Information#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER1
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 06 02 12:00:00:254#+0200#Info#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: Guest
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 06 02 12:05:00:465#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER9
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 06 02 12:50:00:065#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Login
User: USER9
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 24 10:43:38:683#+0200#Info#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103EC9E50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER1#5##C47731E44D00000bae##0#Thread[HTTP Worker [@1473726842],5,Dedicated_Application_Thread]#Plain##
Login
User: USER1
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

logrotate_0.0.log

#2.0^H#2018 05 24 11:05:04:011#+0200#Info#/Sy/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103EC9F50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER4#6##A40B81404D03c0bae##0#Thread[HTTP Worker [@1264376989],5,Dedicated_Application_Thread]#Plain##
Login
User: USER4
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 24 11:04:59:410#+0200#Info#/Sy/Sec/Informtion#
#BC-JAS-SEC#security#C0000A7103EC9F50000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER4#6##A40B81404D03c0bae##0#Thread[HTTP Worker [@1264376989],5,Dedicated_Application_Thread]#Plain##
Login
User: USER4
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 24 11:05:07:100#+0200#Info#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA0C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER3#7##9ACF7Ec0bae##0#Thread[HTTP Worker [@124054179],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER3
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 24 11:07:21:314#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20E0000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#03c0bae##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Login
User: USER2
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 24 11:07:21:314#+0200#Info#/Sys/Sec/Information#
#BC-JAS-SEC#security#C0000A7103ECA20E0000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#03c0bae##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Login
User: USER2
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 26 10:48:02:458#+0200#Warn#/Sys/Sec/Authentication#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER2
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

#2.0^H#2018 05 28 10:00:25:000#+0200#Info#/Sys/Sec/Information#
#BC-JAS-SEC#security#C0000A7103ECA20050000508C#3935150000000004#common.com/irj#com.common.services.security.authentication.logincontext.table#USER2#0##E0E##0#Thread[HTTP Worker [@2033389552],5,Dedicated_Application_Thread]#Plain##
Logout
User: USER0
IP Address: 127.0.0.1
Authentication Stack: ticket
Authentication Stack Properties:

我在使用第 54 行的语句时遇到问题:

#next unless $. > $jumprow;

我认为它不起作用,因为我使用了以下记录分隔符,但我不明白我必须使用哪种分隔符来解决这个问题:

$/ = "\n\n";  # record separator

为了调试代码,我插入了以下语句:

print "next unless $. > $jumprow\n";

如我所见,$。值与文件的行号不同(原因是记录分隔符带有双新行---> $/ = "\n\n";)

如果我删除我的双新行,脚本不起作用

我的脚本的详细信息:(1)第一步:读取 STAT_FILE 以查看上次运行中读取的行

(2)第二步:我将日期、时间、操作(登录或注销)和用户(如果不是访客)封装到一个数组(@events)中。我按用户排序数组(默认不是按日期)。

(3)第三步:我将读取的日志文件的信息打印到 STAT_FILE 中

(4)第四步:我将排序后的@event 数组写入名为 MM-YYYY 的目录中的文件 parse_log.csv 中(这取决于我的事件日期)。

你能帮我找到我的脚本的解决方案吗?

4

1 回答 1

2

我以为我们昨天已经讲过了。

if ( $currentinode == $inode ) {
    # Get rows to jump for this $INPUT
    my $jumprow = $older_count{$INPUT}[0];
}
else {
    # If file has been changed
    my $jumprow = 0;
}

这些块中的每一个都声明了一个 $jumprow变量。当您退出声明它们的块时(即在下一行),这些变量中的每一个都将不复存在。

如果要在if/else块之外访问这些变量,则需要在更高级别声明它们。

my $jumprow;
if ( $currentinode == $inode ) {
    # Get rows to jump for this $INPUT
    $jumprow = $older_count{$INPUT}[0];
}
else {
    # If file has been changed
    $jumprow = 0;
}

或者(更简单):

my $jumprow = 0;
$jumprow = $older_count{$INPUT}[0] if $currentinode == $inode;

或者

my $jumprow = $currentinode == $inode ? $older_count{$INPUT}[0] : 0;
于 2018-08-01T15:50:41.830 回答