16

我有几个包含所有以时间戳开头的行的日志，以便以下工作按预期合并它们：

cat myLog1.txt myLog2.txt | sort -n > combined.txt

问题是，myLog2.txt 也可以包含没有时间戳的行（例如 java 堆栈跟踪）。有没有任何自定义脚本的简单方法仍然可以合并它们并保留多行内容？

示例myLog1.txt

11:48:18.825 [main] INFO  org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
11:48:55.784 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396: Updating schema

示例myLog2.txt

11:48:35.377 [qtp1484319352-19] ERROR c.w.b.c.ControllerErrorHandler -
org.springframework.beans.TypeMismatchException: Failed to convert value of type   'java.lang.String' to required type 'org.joda.time.LocalDate'; nested exception is    org.springframework.core.convert.ConversionFailedException: Failed to convert from type     java.lang.String to type @org.springframework.web.bind.annotation.RequestParam   @org.springframework.format.annotation.DateTimeFormat org.joda.time.LocalDate for value    '[2013-03-26]'; nested exception is java.lang.IllegalArgumentException: Invalid format: "    [2013-03-26]"
    at org.springframework.beans.TypeConverterSupport.doConvert(TypeConverterSupport.java:68) ~[spring-beans-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.beans.TypeConverterSupport.convertIfNecessary(TypeConverterSupport.java:45) ~[spring-beans-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.validation.DataBinder.convertIfNecessary(DataBinder.java:595) ~[spring-context-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:98) ~[spring-web-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:77) ~[spring-web-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:162) ~[spring-web-3.2.1.RELEAS

预期产出

11:48:18.825 [main] INFO  org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
11:48:35.377 [qtp1484319352-19] ERROR c.w.b.c.ControllerErrorHandler -
org.springframework.beans.TypeMismatchException: Failed to convert value of type   'java.lang.String' to required type 'org.joda.time.LocalDate'; nested exception is    org.springframework.core.convert.ConversionFailedException: Failed to convert from type     java.lang.String to type @org.springframework.web.bind.annotation.RequestParam   @org.springframework.format.annotation.DateTimeFormat org.joda.time.LocalDate for value    '[2013-03-26]'; nested exception is java.lang.IllegalArgumentException: Invalid format: "    [2013-03-26]"
    at org.springframework.beans.TypeConverterSupport.doConvert(TypeConverterSupport.java:68) ~[spring-beans-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.beans.TypeConverterSupport.convertIfNecessary(TypeConverterSupport.java:45) ~[spring-beans-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.validation.DataBinder.convertIfNecessary(DataBinder.java:595) ~[spring-context-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:98) ~[spring-web-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:77) ~[spring-web-3.2.1.RELEASE.jar:3.2.1.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:162) ~[spring-web-3.2.1.RELEAS
11:48:55.784 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396: Updating schema

谢谢马可

4

5 回答 5

24

I was struggling with the same issue and finally I think I've got it. Try do it like:

sort -nbms -k1.1,1.2 -k1.4,1.5 -k1.7,1.8 -k1.10,1.12 myLog1.txt myLog2.txt > combined.txt

It's still not fully clear to myself, I'll try to give some explanation though. According to the man pages used switches mean:

-n, --numeric-sort - compare according to string numerical value.

-b, --ignore-leading-blanks - ignore leading blanks.

-s, --stable - stabilize sort by disabling last-resort comparison

-m, --merge - merge already sorted files; do not sort

-k, --key=POS1[,POS2] - start a key at POS1 (origin 1), end it at POS2 (default end of line)

  • log files are already ordered so we don't need to sort them again, only determine which line goes where upon merging. That's why -m. It's crucial to keep stacktraces from getting scrambled.
  • -b is not necessary in this case as somehow -n and -m combined keeps stacktrace lines from getting clustered. I left it just in case as most of stacktrace lines starts with blanks.
  • -n apparently stops comparing key whenever there is a non-numeric character in the key. That's the second crucial bit for keeping stacktraces in place. Important is if it was -n -k1,1 it would only sort the log files by hour as colon is non-numeric. Apart from that -n speeds up numeric comparison so we would like to have it anyway.
  • the problem mentioned in the previous point is solved by pointing to specific characters positions in each key, that's why -k1.1,1.2 (first and second digit of hour) -k1.4,1.5 (first and second digit of minutes) and so on. The first digit before the dot is always '1' as it points to the first column of the file line (which in our case is time). Shortly it's -kA,B where A and B are column positions in a given line (by default lines are delimited by blanks). Format of A and B used is .. Keep in mind that whenever there is a non-numeric character between A and B everything after it will be ignored in comparison if -n used.
  • -s disables default behaviour which is: whenever keys by which comparison is being done are the same full string comparison of the lines is done. We don't want that to preserve original log entries order. Not sure if it's necessary with -m though.
于 2014-11-26T13:54:09.370 回答
1

不——不能用一个简单的命令 IMMHO 来完成。

但是 - 这是一个脚本来做到这一点（这是一个挑战......）

@ECHO OFF
SETLOCAL
:: First log to tempfile
COPY /y mylog.txt "%temp%\combinedlogs.tmp" >NUL
(
FOR /f "delims=" %%i IN (mylog2.txt) DO (
 SET line=%%i
 ECHO %%i|FINDSTR /b /r "[012][0-9]:[0-5][0-9]:[0-5][0-9]\.[0-9][0-9][0-9]" >NUL
 IF ERRORLEVEL 1 (
  SETLOCAL ENABLEDELAYEDEXPANSION
 ECHO(!stamp:~0,12!!count!!line!
  ENDLOCAL
  SET /a count+=1
 ) ELSE (
 SET /a count=100
 ECHO %%i
 SET stamp=%%i
 )
)
)>>"%temp%\combinedlogs.tmp"
(
FOR /f "delims=" %%i IN ('SORT "%temp%\combinedlogs.tmp"') DO (
 SET line=%%i
 SETLOCAL enabledelayedexpansion
 IF "!line:~12,1!"==" " (ECHO(%%i) ELSE (ECHO(!line:~15!)
 ENDLOCAL
)
)>combinedlogs.txt
DEL "%temp%\combinedlogs.tmp" /F /Q

将带有所有时间戳条目的第一个日志复制到临时
文件 通过以下方式处理第二个文件

  • 直接输出任何带时间戳的行，保存时间戳行并设置一个 3 位计数器
  • 为其他行输出印章部分+计数器+原始文本并碰撞计数器

因此 Tempfile 是

Timestamp1 line1 from file1
..
Timestampn linen from file1
timestampA line1 from file2 with timestamp
timestampA100 UNtimestamped line2from file2
timestampA101 UNtimestamped line3from file2
timestampB line4 from file2 with timestamp
timestampB100 UNtimestamped line5from file2
timestampB101 UNtimestamped line6from file2
...

对结果进行排序并重新处理
第 13 个字符中没有空格的行是第二个文件中没有时间戳的行，因此

  • 输出除前 15 个字符以外的所有字符（时间戳 12 个字符 + 3 个用于计数器）
  • 否则，时间戳行，所以输出所有。

完毕！

于 2013-04-08T03:50:54.410 回答
1

这是在 bash shell 中简单合并文件的一种方法（而不是昂贵的手段——因为日志文件已经排序）。这对于数百兆字节或更多的大文件很重要，就像现实世界的日志文件一样。

此解决方案假定您的日志中没有 NUL 字节，这对于我遇到的每个日志文件都是如此，具有各种字符集。

基本思想：

  1. 通过在每个输入文件中用 NUL 替换这些换行符，将所有多行连接到单行
  2. 对替换的文件执行 asort -m以合并它们
  3. 将 NUL 替换回合并结果上的换行符

由于第一步多次完成，我给它起了一个别名：

alias a="awk '{ if (match(\$0, /^[0-9]{2}:[0-9]{2}:[0-9]{2}\\./, _))\
{ if (NR == 1) printf \"%s\", \$0; else printf \"\\n%s\", \$0 }\
else printf \"\\0%s\", \$0 } END { print \"\" }'"

这是执行所有 3 个步骤的命令：

sort -m <(a myLog1.txt) <(a myLog2.txt) | tr '\0' '\n'

有关更多信息，请参阅https://superuser.com/a/838446/125379

于 2014-11-12T23:07:40.933 回答
0

您应该在日志文件中使用merge、stable、ignore-leading-blanks、numeric-sort和易于排序的日期时间格式（例如yyyyMMddHHmmssSSS）。

因此，我将您的日志格式更改为更易于排序，结果是sort -bsnm log1 log2：

 $ cat -n log1 log2 && sort -m -b -n -s log1 log2
      1 114818825 [main] INFO  org.hibernate.cfg.Environment - HHH000206 hibernate.properties not found
      2 114855784 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396 Updating schema
      1 114835377 [qtp1484319352-19] ERROR c.w.b.c.ControllerErrorHandler -
      2 org.springframework.beans.TypeMismatchException Failed to convert value of type   'java.lang.String' to required type 'org.joda.time.LocalDate'; nested exception is    org.springframework.core.convert.ConversionFailedException Failed to convert from type     java.lang.String to type @org.springframework.web.bind.annotation.RequestParam   @org.springframework.format.annotation.DateTimeFormat org.joda.time.LocalDate for value    '[2013-03-26]'; nested exception is java.lang.IllegalArgumentException Invalid format "    [2013-03-26]"
      3     at org.springframework.beans.TypeConverterSupport.doConvert(TypeConverterSupport.java68) ~[spring-beans-3.2.1.RELEASE.jar3.2.1.RELEASE]
      4 at org.springframework.beans.TypeConverterSupport.convertIfNecessary(TypeConverterSupport.java45) ~[spring-beans-3.2.1.RELEASE.jar3.2.1.RELEASE]
      5 at org.springframework.validation.DataBinder.convertIfNecessary(DataBinder.java595) ~[spring-context-3.2.1.RELEASE.jar3.2.1.RELEASE]
      6 at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java98) ~[spring-web-3.2.1.RELEASE.jar3.2.1.RELEASE]
      7 at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java77) ~[spring-web-3.2.1.RELEASE.jar3.2.1.RELEASE]
      8 at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java162) ~[spring-web-3.2.1.RELEAS
      9 
 114818825 [main] INFO  org.hibernate.cfg.Environment - HHH000206 hibernate.properties not found
 114835377 [qtp1484319352-19] ERROR c.w.b.c.ControllerErrorHandler -
 org.springframework.beans.TypeMismatchException Failed to convert value of type   'java.lang.String' to required type 'org.joda.time.LocalDate'; nested exception is    org.springframework.core.convert.ConversionFailedException Failed to convert from type     java.lang.String to type @org.springframework.web.bind.annotation.RequestParam   @org.springframework.format.annotation.DateTimeFormat org.joda.time.LocalDate for value    '[2013-03-26]'; nested exception is java.lang.IllegalArgumentException Invalid format "    [2013-03-26]"
     at org.springframework.beans.TypeConverterSupport.doConvert(TypeConverterSupport.java68) ~[spring-beans-3.2.1.RELEASE.jar3.2.1.RELEASE]
 at org.springframework.beans.TypeConverterSupport.convertIfNecessary(TypeConverterSupport.java45) ~[spring-beans-3.2.1.RELEASE.jar3.2.1.RELEASE]
 at org.springframework.validation.DataBinder.convertIfNecessary(DataBinder.java595) ~[spring-context-3.2.1.RELEASE.jar3.2.1.RELEASE]
 at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java98) ~[spring-web-3.2.1.RELEASE.jar3.2.1.RELEASE]
 at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java77) ~[spring-web-3.2.1.RELEASE.jar3.2.1.RELEASE]
 at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java162) ~[spring-web-3.2.1.RELEAS

 114855784 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396 Updating schema

正如@Magoo 的回答中所说，您的日志日期时间当前的格式很难排序。

于 2014-01-08T17:54:57.290 回答
0

The open source tool (Java GitHub) lets you combine log files with different formats including multilines into a merged file.

The tool allows to shift the time of records in a log file. It can be useful when files come from different time zones.

It also allows to generate additional information to a merged file, for example applications names or timestamps in a uniform format, see the example.

The tool can be used as a command line tool or Java library. Note: I'm the author.

于 2020-01-26T10:56:23.990 回答