2

我对 Kibana 和 Elastic 搜索的使用非常陌生,但我可以使用它并且可以搜索字符串等没问题。

但我想搜索这个正则表达式

threshold of 1000 ms: ([5-9],\d+|[1-9][0-9],\d+|[1-9][0-9][0-9],\d+)

使用https://regex101.com/并使用 Notepad++ 进行测试。但无论我尝试什么,我都无法让它在 Kibana 中运行。

我尝试将索引字段更改为关键字而不是文本,但仍然不起作用。

即使是一个简单的正则表达式

message:/192.168.99.[0-9]{3}/

不像你期望的那样工作。

有人可以建议一种方法来让它工作吗?它快把我逼疯了。

我应该提到我正在使用 Filebeat 6.4.2、Elasticsearch 6.4.2 和 Kibana 6.4.2

谢谢

映射

"mappings": {
  "doc": {
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "long"
      },
      "CorrelationId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "InteractionNumber": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "RequestorId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "alertType": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "app": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "beat": {
        "properties": {
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "input": {
        "properties": {
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "level": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "logger_name": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "message": {
        "type": "keyword"
      },
      "nodeId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "offset": {
        "type": "long"
      },
      "pegathread": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "prospector": {
        "properties": {
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "source": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "source_host": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-env": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-node": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-vm": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "stack": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "tenantid": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "thread_name": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "userid": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "x-requested-with": {
        "type": "keyword",
        "ignore_above": 1024
      }
    }
  }
}

示例消息字段内容

2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*

将消息类型添加为字符串,这是 _analyze 结果:-

{
  "token": "threshold",
  "start_offset": 1276,
  "end_offset": 1285,
  "type": "<ALPHANUM>",
  "position": 128
},
{
  "token": "of",
  "start_offset": 1286,
  "end_offset": 1288,
  "type": "<ALPHANUM>",
  "position": 129
},
{
  "token": "1000",
  "start_offset": 1289,
  "end_offset": 1293,
  "type": "<NUM>",
  "position": 130
},
{
  "token": "ms",
  "start_offset": 1294,
  "end_offset": 1296,
  "type": "<ALPHANUM>",
  "position": 131
},
{
  "token": "8088",
  "start_offset": 1298,
  "end_offset": 1302,
  "type": "<NUM>",
  "position": 132
},
{
  "token": "ms",
  "start_offset": 1303,
  "end_offset": 1305,
  "type": "<ALPHANUM>",
  "position": 133
}
4

1 回答 1

2

在版本 6.2.4 上测试

添加了以下带有映射的索引,如下所示

    PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "message": {
          "type": "text"
        },
        "message2": {
          "type": "keyword"
        }
      }
    }
  }
}

将 2 个文档添加到索引中,如下所示

PUT test/_doc/1?refresh
{
  "message": "hellothere",
  "message2":"2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*" 
} 


PUT test/_doc/2?refresh
{
  "message": "2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*",
  "message2":"hellothere" 
}

在 0 个结果中搜索message2: /192.168.99.[0-9]{3}/结果

message: /192.168.99.[0-9]{3}/在 doc#2 中搜索结果

message2: /.*192.168.99.[0-9]{3}.*/在 doc#1 中搜索结果

在 0 个结果中搜索message: /pegarules.session/结果。

但是在 doc#1 中搜索message: /.*pegarules.session.*/结果,因为倒排索引有"token": "com.pega.pegarules.session.internal.engineinterface.service.httpapi"

message2: /.*pegarules.session.*/在 doc#1` 中搜索结果

因此,消息字段(类型text)被标记化,并且正则表达式搜索通配符标记模式正在返回结果。

其中,message2 字段(type keyword)不被分析,而是按原样放入倒排索引中。192.168.99.[0-9]{3}除非我们添加贪婪量词 (.*),否则正则表达式搜索模式不会返回任何内容

Lucene 正则表达式引擎与 Perl 不兼容,但支持较小范围的运算符,因此它可能无法像正则表达式一样工作和匹配结果。

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html#regexp-syntax

于 2018-10-23T07:56:08.703 回答