0

Splunk 版本 6.5.3.1

Splunk 构建 bf0ff7c2ab8b

詹金斯版本1.642.3 或 2.32.3

在每个 Jenkins master 上，都有一个 splunk 进程在运行。

$ ps -eAf|grep splunk
splunk    58877      1 20 Feb16 ?        42-23:27:37 splunkd -p 8089 restart
splunk    58878  58877  0 Feb16 ?        00:00:00 [splunkd pid=58877] splunkd -p 8089 restart [process-runner]
asangal   91197  91175  0 12:38 pts/2    00:00:00 grep --color=auto splunk

Splunk 进程监控/扫描log文件以查找我们实例中的任何 Jenkins 作业，即$JENKINS_HOME/jobs/<JOB_NAME>/builds/<BUILD_NUMBER>/log文件下。

$ pwd
/opt/splunkforwarder/etc/system/local
$ cat inputs.conf
[default]
host = jenkins-master-project-prod-1-609 

[monitor:///var/log/jenkins]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True

[monitor:///home/jenkins/jobs/.../builds/.../log]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True
crcSalt = <SOURCE>

...
.....
... more config code here ...
....
..

在 Splunk GUI 中，当我运行一个简单的查询来查找 Splunk 捕获的任何内容index并且来自任何source（文件）时，我确实看到了有效的输出。注意：实际行输出被截断。正如您所看到的条形图，数据在那里并且表格被填充。 在此处输入图像描述

在我的 Jenkins 工作中，我有时会收到一些警告、信息、错误（我已经Log Parser Plugin在 J​​enkins 级别使用），我正在尝试编写一个脚本，最后从 Splunk 获取 Jenkins 工作的这个 LOG 输出15、30 分钟或过去 1-7 小时或 1-30 天，并找出在给定时间段内发现了多少警告、错误等（基于某些关键字、正则表达式）。注意：有各种这样的 Jenkins master，Splunk 在其中运行，我的目标是与 Splunk 交谈并获取我需要的数据（而不是与 500 个 Jenkins master 交谈）。

我尝试了以下CURL命令，这些命令返回给我一个 SEARCH ID，​​但它没有做任何事情。

在下面的 CURL 命令中，我传递了一个更精细的查询来获取数据。我说在最后 30 分钟内获取 Splunk 拥有的所有信息（可以根据 GUI 添加字段）以及日志的位置和位置：（index第some-jenkins-prod-index-in-splunk一个是作业名称，第二个是内部版本号）然后我说在 splunk 中搜索日志，其中日志包含行/关键字/正则表达式（如下所列使用条件），并以 JSON 格式显示输出。source/home/jenkins/jobs/*/builds/*/log**OR

➜  ~ p=$(cat ~/AKS/rOnly/od.p.txt)
➜  ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"messages":[{"type":"ERROR","text":"Method Not Allowed"}]}%                                                                                  ➜  ~

如您所见，它给了我不允许的方法。

当我在 URL 部分给出以下查询时/jobs，我得到了一个有效的 SEARCH ID。 

➜  ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"sid":"1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A"}%

使用此搜索 ID，我试图访问主日志，但它不起作用。我jq用来过滤 JSON 输出以显示在一个漂亮的布局中。

 ➜  ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .
{
  "links": {},
  "origin": "http://splunk.server.mycompany.com/services/search/jobs",
  "updated": "2017-09-15T09:44:33-07:00",
  "generator": {
    "build": "bf0ff7c2ab8b",
    "version": "6.5.3.1"
  },
  "entry": [
    {
      "name": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
      "id": "http://splunk.server.mycompany.com/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
      "updated": "2017-09-15T09:44:33.942-07:00",
      "links": {
        "alternate": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
        "search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
        "events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
        "results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
        "results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
        "timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
        "summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
        "control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"
      },
      "published": "2017-09-15T09:43:59.000-07:00",
      "author": "my_splunk_user",
      "content": {
        "bundleVersion": "17557160226808436058",
        "canSummarize": false,
        "cursorTime": "1969-12-31T16:00:00.000-08:00",
        "defaultSaveTTL": "2592000",
        "defaultTTL": "600",
        "delegate": "",
        "diskUsage": 561152,
        "dispatchState": "DONE",
        "doneProgress": 1,
        "dropCount": 0,
        "earliestTime": "2017-09-15T09:13:58.000-07:00",
        "eventAvailableCount": 0,
        "eventCount": 30,
        "eventFieldCount": 0,
        "eventIsStreaming": true,
        "eventIsTruncated": true,
        "eventSearch": "search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" ",
        "eventSorting": "none",
        "isBatchModeSearch": true,
        "isDone": true,
        "isEventsPreviewEnabled": false,
        "isFailed": false,
        "isFinalized": false,
        "isPaused": false,
        "isPreviewEnabled": false,
        "isRealTimeSearch": false,
        "isRemoteTimeline": false,
        "isSaved": false,
        "isSavedSearch": false,
        "isTimeCursored": true,
        "isZombie": false,
        "keywords": "\"*: binary operator expected*\" \"*: no such file or directory*\" \"*: too many arguments*\" \"*npm warn retry*\" \"*rsync: failed to set times on*\" \"*svn: e200033: *\" \"*warning: *\" earliest::-30m index::some-jenkins-prod-index-in-splunk source::/home/jenkins/jobs/*/builds/*/log",
        "label": "",
        "latestTime": "2017-09-15T09:43:59.561-07:00",
        "normalizedSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex source=\".*/[0-9][0-9]*/log\" | fields keepcolorder=t \"_raw\" \"host\" \"source\"",
        "numPreviews": 0,
        "optimizedSearch": "| search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
        "pid": "2174",
        "priority": 5,
        "remoteSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex  source=\".*/[0-9][0-9]*/log\"  | fields  keepcolorder=t \"_raw\" \"host\" \"source\"",
        "reportSearch": "table  host, source, _raw",
        "resultCount": 30,
        "resultIsStreaming": false,
        "resultPreviewCount": 30,
        "runDuration": 0.579,
        "sampleRatio": "1",
        "sampleSeed": "0",
        "scanCount": 301,
        "searchCanBeEventType": false,
        "searchEarliestTime": 1505492038,
        "searchLatestTime": 1505493839.21872,
        "searchTotalBucketsCount": 37,
        "searchTotalEliminatedBucketsCount": 0,
        "sid": "1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
        "statusBuckets": 0,
        "ttl": 600,
        "performance": {
          "command.fields": {
            "duration_secs": 0.035,
            "invocations": 48,
            "input_count": 30,
            "output_count": 30
          },
          "command.regex": {
            "duration_secs": 0.048,
            "invocations": 48,
            "input_count": 30,
            "output_count": 30
          },
          "command.search": {
            "duration_secs": 1.05,
            "invocations": 48,
            "input_count": 0,
            "output_count": 30
          },
          "command.search.calcfields": {
            "duration_secs": 0.013,
            "invocations": 16,
            "input_count": 301,
            "output_count": 301
          },
          "dispatch.optimize.reparse": {
            "duration_secs": 0.001,
            "invocations": 1
          },
          "dispatch.optimize.toJson": {
            "duration_secs": 0.001,
            "invocations": 1
          },
          "dispatch.optimize.toSpl": {
            "duration_secs": 0.001,
            "invocations": 1
          },
          "dispatch.parserThread": {
            "duration_secs": 0.048,
            "invocations": 48
          },
          "dispatch.reduce": {
            "duration_secs": 0.001,
            "invocations": 1
          },
          "dispatch.stream.remote": {
            "duration_secs": 1.05,
            "invocations": 48,
            "input_count": 0,
            "output_count": 332320
          },
          "dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8081": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8082": {
            "duration_secs": 0.001,
            "invocations": 1,
            "input_count": 0,
            "output_count": 5422
          },
          "dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8081": {
            "duration_secs": 0.058,
            "invocations": 2,
            "input_count": 0,
            "output_count": 16948
          },
          "dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8082": {
            "duration_secs": 0.066,
            "invocations": 2,
            "input_count": 0,
            "output_count": 14415
          },
          "dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8081": {
            "duration_secs": 0.059,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15858
          },
          "dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8082": {
            "duration_secs": 0.065,
            "invocations": 2,
            "input_count": 0,
            "output_count": 11867
          },
          "dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8081": {
            "duration_secs": 0.061,
            "invocations": 2,
            "input_count": 0,
            "output_count": 20695
          },
          "dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8082": {
            "duration_secs": 0.06,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15193
          },
          "dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8081": {
            "duration_secs": 0.063,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15932
          },
          "dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8082": {
            "duration_secs": 0.064,
            "invocations": 2,
            "input_count": 0,
            "output_count": 14415
          },
          "dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8081": {
            "duration_secs": 0.061,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15418
          },
          "dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8082": {
            "duration_secs": 0.058,
            "invocations": 2,
            "input_count": 0,
            "output_count": 11866
          },
          "dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8081": {
            "duration_secs": 0.075,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15661
          },
          "dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8082": {
            "duration_secs": 0.071,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15845
          },
          "dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8081": {
            "duration_secs": 0.066,
            "invocations": 2,
            "input_count": 0,
            "output_count": 14406
          },
          "dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8082": {
            "duration_secs": 0.072,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15524
          },
          "dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8081": {
            "duration_secs": 0.067,
            "invocations": 2,
            "input_count": 0,
            "output_count": 16009
          },
          "dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8082": {
            "duration_secs": 0.068,
            "invocations": 2,
            "input_count": 0,
            "output_count": 15516
          },
          "dispatch.writeStatus": {
            "duration_secs": 0.012,
            "invocations": 7
          },
          "startup.configuration": {
            "duration_secs": 2.045,
            "invocations": 33
          },
          "startup.handoff": {
            "duration_secs": 14.595,
            "invocations": 33
          }
        },
        "messages": [
          {
            "type": "INFO",
            "text": "Your timerange was substituted based on your search string"
          },
          {
            "type": "WARN",
            "text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 using the uri-scheme=http because peer has status=\"Down\".  Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
          },
          {
            "type": "WARN",
            "text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 using the uri-scheme=http because peer has status=\"Down\".  Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
          }
        ],
        "request": {
          "search": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw"
        },
        "runtime": {
          "auto_cancel": "0",
          "auto_pause": "0"
        },
        "searchProviders": [
          "mr11p01if-ztbv02090901.mr.if.mycompany.com-8081",
          "mr11p01if-ztbv16142101.mr.if.mycompany.com-8082",
          "mr11p01if-ztbv16142301.mr.if.mycompany.com-8081",
          "mr11p01if-ztbv16142301.mr.if.mycompany.com-8082",
          "mr21p01if-ztbv14080101.mr.if.mycompany.com-8081",
          "mr21p01if-ztbv14080101.mr.if.mycompany.com-8082",
          "mr22p01if-ztbv07132101.mr.if.mycompany.com-8081",
          "mr22p01if-ztbv07132101.mr.if.mycompany.com-8082",
          "mr22p01if-ztbv09013201.mr.if.mycompany.com-8081",
          "mr22p01if-ztbv09013201.mr.if.mycompany.com-8082",
          "mr90p01if-ztep02103701.mr.if.mycompany.com-8081",
          "mr90p01if-ztep02103701.mr.if.mycompany.com-8082",
          "mr90p01if-ztep04044101.mr.if.mycompany.com-8081",
          "mr90p01if-ztep04044101.mr.if.mycompany.com-8082",
          "mr90p01if-ztep06024101.mr.if.mycompany.com-8081",
          "mr90p01if-ztep06024101.mr.if.mycompany.com-8082",
          "mr90p01if-ztep12023601.mr.if.mycompany.com-8081",
          "mr90p01if-ztep12023601.mr.if.mycompany.com-8082",
          "mr90p01if-ztep12043901.mr.if.mycompany.com-8081",
          "mr90p01if-ztep12043901.mr.if.mycompany.com-8082",
          "pv31p01if-ztbv08050801.pv.if.mycompany.com-8081",
          "pv31p01if-ztbv08050801.pv.if.mycompany.com-8082",
          "pv31p01if-ztbv08051001.pv.if.mycompany.com-8081",
          "pv31p01if-ztbv08051001.pv.if.mycompany.com-8082",
          "pv31p01if-ztbv08051201.pv.if.mycompany.com-8081",
          "pv31p01if-ztbv08051201.pv.if.mycompany.com-8082"
        ]
      },
      "acl": {
        "perms": {
          "read": [
            "my_splunk_user"
          ],
          "write": [
            "my_splunk_user"
          ]
        },
        "owner": "my_splunk_user",
        "modifiable": true,
        "sharing": "global",
        "app": "search",
        "can_write": true,
        "ttl": "600"
      }
    }
  ],
  "paging": {
    "total": 1,
    "perPage": 0,
    "offset": 0
  }
}
➜  ~
➜  ~

但是，如您所见，生成的 JSON 输出没有用，因为它没有显示或包含我可以使用的任何 Jenkins 作业的输出。

如果在 CURL 命令中，对于 Splunk URL，如果我尝试以下任何 URL 端点，它会给我一个错误。

    "search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
    "events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
    "results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
    "results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
    "timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
    "summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
    "control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"

例如：如果我尝试URL.../<SEARCH_ID>/events, orURL/.../<SEARCH_ID>/results等​​，我会收到以下错误。

curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .

{
  "messages": [
    {
      "type": "FATAL",
      "text": "Method Not Allowed"
    }
  ]
}

我正在尝试查找最近 N 个时间段内的主机名、源（Jenkins 作业日志的路径）、实际作业的控制台输出（我可以读取和解析以生成有意义的信息）、有多少、错误、警告、奇怪线条出现了，根据一些阈值，如果数字超过这些阈值，那么我需要发送电子邮件通知。

我可以编写所有这些代码，但我没有解决这里的第一个难题，即让 Splunk 吐出 Jenkins 作业的 CONSOLE OUTPUT，splunk 正在监视文件系统。

最终目标是将有意义的数据转储到表单或 JSON 或 CSV 的文本文件中，并将该数据转换为一些有意义的条形图/饼图等。

例如：如果 data.csv 包含：

age,population
<5,2704659
5-13,4499890
14-17,2159981
18-24,3853788
25-44,14106543
45-64,8819342
65-85,312463
≥85,81312463

然后使用以下文件，我可以将此原始数据转换为饼图，如下图所示。

<!DOCTYPE html>
<meta charset="utf-8">
<style>

.arc text {
  font: 10px sans-serif;
  text-anchor: middle;
}

.arc path {
  stroke: #fff;
}

</style>
<svg width="960" height="500"></svg>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script>

var svg = d3.select("svg"),
    width = +svg.attr("width"),
    height = +svg.attr("height"),
    radius = Math.min(width, height) / 2,
    g = svg.append("g").attr("transform", "translate(" + width / 2 + "," + height / 2 + ")");

var color = d3.scaleOrdinal(["#98abc5", "#8a89a6", "#7b6888", "#6b486b", "#a05d56", "#d0743c", "#ff8c00"]);

var pie = d3.pie()
    .sort(null)
    .value(function(d) { return d.population; });

var path = d3.arc()
    .outerRadius(radius - 10)
    .innerRadius(0);

var label = d3.arc()
    .outerRadius(radius - 40)
    .innerRadius(radius - 40);

d3.csv("data.csv", function(d) {
  d.population = +d.population;
  return d;
}, function(error, data) {
  if (error) throw error;

  var arc = g.selectAll(".arc")
    .data(pie(data))
    .enter().append("g")
      .attr("class", "arc");

  arc.append("path")
      .attr("d", path)
      .attr("fill", function(d) { return color(d.data.age); });

  arc.append("text")
      .attr("transform", function(d) { return "translate(" + label.centroid(d) + ")"; })
      .attr("dy", "0.35em")
      .text(function(d) { return d.data.age; });
});

</script>

生成的饼图（由于 csv 文件和 html 文件）： 在此处输入图像描述

4

2 回答 2

0

找到了解决方案。

我只需要使用services/search/jobs/export端点。

让我们找出我们的 Jenkins host（其中包含 Jenkins 作业）是什么，作业的名称是什么（可以从source日志文件的路径中解析/grep）以及 Jenkins 作业的实际控制台输出（_raw）是什么。此外，让我们限制我们在过去 30 分钟内查找数据的搜索（即earliest=-30m）。

实际上有3种方法可以做到这一点。

1）通过在命令行传递用户名和密码。

2）通过生成一个SESSION TOKEN我们可以在标题中的任何未来 CURL 命令中使用的。

3）通过生成一个--cookie "${COOKIE}"ID并使用它。这是一种首选方法，因为它将 cookie 值复制到 Splunk 使用的任何后端服务器。要使用的 Cookie 名称：splunkd_8081

最后两种解决方案取决于第一种使用用户凭据创建 SESSION 或 COOKIE ID 的方法。


解决方案1：

1) 这里我们将使用我们的 splunk 服务器

2）在命令行中传递用户名和密码

3) 提供用于查找/获取 Splunk 数据的 Splunk 选项（用于包含特定行的 Jenkins 日志）并进行一些额外的正则表达式匹配（以便它为源路径返回准确的 Jenkins build#，而不是source为相同的控制台输出。Jenkins latestBuild、latestSuccessfulBuild 等是指向编号构建的符号链接，我们不想source在输出中列出这些符号链接条目，所以我使用正则表达式模式来查找source它包含 NUMBERED 构建的路径#就在log源路径中的文件之前）。

4) 然后，我|只过滤掉 3 个字段：host、source和_raw（Splunk 会返回）。host包含，詹金斯服务器有詹金斯的工作。source在它的值中包含 Jenkins 作业名称、build# 等信息。_raw变量包含 Jenkins 作业的控制台输出（几行接近我们试图在 Jenkins 作业的控制台输出中搜索的字符串/行）。

注意：所有这 3 个字段都在字典变量中可用，result所以我只是输出它。

5）然后，我提供json格式的输出（你也可以使用csv）。最后我jq用来过滤信息。

注意：如果您使用jq -r ".result._raw"（即_raw字典变量中的字段，result那么它将为您提供控制台输出的逐行输出（而不是给您一个\n嵌入其中的 blob）。您也可以使用 sed 's/\\n/\n/g'但jq -r ".result._raw"很容易）。

运行的命令：

$ p="$(cat ~/my_secret_password.txt)"
$
$ # The above command will set my password in variable 'p'
$
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"
$
$ # The following will give you LINE by LINE output for the console output 
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq -r ".result._raw"


注意：用户 ID 和密码作为（在和实际 splunk 用户名-umy_splunk_user:$p之后/之间不需要空格）传递。-u

解决方案2：

解决方案编号 2 是通过使用会话密钥/ID。您首先必须使用services/auth/login端点。

要生成会话密钥/ID，请运行以下命令。

注意：要生成 SESSION 密钥，您确实需要先提供您的凭据，但在以后的 CURL/API 调用/命令中，您可以只在标头中传递 SESSION 密钥。

1) 生成会话密钥/id。

$ p=$(cat ~/my_secret_password.txt)
$ curl -k https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=my_splunk_userid --data-urlencode password=$p 
<response>
  <sessionKey>192fd3e46a31246da7ea7f109e7f95fd</sessionKey>
</response>

2) 以后在后续搜索中使用会话密钥/ID。

在后续请求中，将标头授权值设置为会话密钥 ()，现在您无需使用-uYourUserID:YourPassword.

$ curl -k -H "Authorization: Splunk 192fd3e46a31246da7ea7f109e7f95fd" --connect-time 10 --max-time 900 https://splunk.mycompany.com:8089/services/search/jobs/export --data search='search earliest=-30m index=some-jenkins-prod-index  source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"


笔记：

1) 用于控制台输出的逐行输出。利用：jq -r ".result._raw"

2）对于找到的搜索计数，您可以使用| stats count

现在，我可以以 CSV 或 JSON 格式提供我需要的数据，并使用绘图功能通过有意义的图表显示数据，或者在阈值大于或小于给定/预期值时发送电子邮件通知（根据我的自动化脚本）。

有关更多信息，请参阅此处的搜索端点的 Splunk REST API 文档：http: //docs.splunk.com/Documentation/Splunk/6.6.3/RESTREF/RESTsearch 和 https://docs.splunk.com/Documentation/Splunk /6.5.3/SearchReference/SearchTimeModifiers

 second: s, sec, secs, second, seconds
 minute: m, min, minute, minutes
 hour: h, hr, hrs, hour, hours
 day: d, day, days
 week: w, week, weeks
 month: mon, month, months
 quarter: q, qtr, qtrs, quarter, quarters
 year: y, yr, yrs, year, years

如果您想搜索截至该点之前的最后 30 天和 30 天的数据，您需要earliest=-60d latest=-30d

解决方案#3：

1）创建COOKIE ID，运行以下命令。

curl -sSv https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=your_splunk_userid --data-urlencode password=your_splunk_secret_password -o /dev/null -d cookie=1 2>&1

它会吐出类似的东西：

< Set-Cookie: splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE; Path=/; HttpOnly; Max-Age=28800; Expires=Wed, 20 Sep 2017 00:23:39 GMT

现在获取值部分< Set-Cookie: <VALUE_upto_the_semi_colon>并将其存储在变量中。IE

export COOKIE="splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE"

2) 现在，在 CURL 命令中使用 cookie 来运行类似的查询，就像我们上面所做的那样。您现在不需要传递凭据-uYourUserID:Password。

$ curl -k --cookie "${COOKIE}" --connect-time 10 --max-time 900 ... rest of the command here similar to examples shown above ... ...
于 2017-09-18T17:59:50.540 回答
0

为了更好地实施，请执行以下操作：

  1. Jenkins 的 Splunk 插件： https ://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins

  2. Splunk 的 Jenkins 插件/应用程序：https ://splunkbase.splunk.com/app/3332/

于 2018-09-14T23:07:28.997 回答