0

附加信息：

  • Logstash 版本：6.3.1
  • 操作系统：macOS 10.13.4
  • 弹性搜索：弹性云上的 6.2.24
  • Kibana：弹性云上的 6.2.24

问题：

你好，

我正在尝试将数据从 logstash 发送到 Elastic Cloud，但是当 logstash 运行时我收到以下错误：

"Empty configuration for pipeline_id: artist_profile_views"

但是，如果我尝试在 logststah.yml 中不定义 xpack 配置的情况下运行 logstash，这可以正常工作，并且 stdout {} 将收集到的数据发送到输出。

请查看我的配置文件：

日志存储.yml

cloud.id: "[...]"
cloud.auth: "[user]:[password]"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: https://elasticCloudUrl
xpack.monitoring.elasticsearch.username: [user]
xpack.monitoring.elasticsearch.password: [password]
xpack.management.enabled: true
xpack.management.pipeline.id: ["artist_profile_views", "searched"]
xpack.management.elasticsearch.username: [user]
xpack.management.elasticsearch.password: [password]
xpack.management.elasticsearch.url: ["https://elasticCloudUrl"]

管道.yml

- pipeline.id: artist_profile_views
  path.config: "pipelines/artist_profile_views.conf"

- pipeline.id: searched
  path.config: "pipelines/searched.conf"

艺术家个人资料视图

input { 
    file {
        path => "/Users/zabaala/Sites/cna/stats/artist_profile_views/artist_profile_views_*.log"
        codec => json
        start_position => "beginning"
    }
}

filter {
    geoip {
        source => "[ip]"
    }   
    useragent {
        source => "[headers][user_agent]"
        target => "[headers][request]"
    }
    mutate {
        remove_field => ["[headers][user_agent]"]
    }
}


output {
    elasticsearch { 
        # hosts => ["https://ElasticCloudUrl"] 
        index => "stats"
    }
    stdout { 
        codec => rubydebug
    }
}

样本数据：

{"artist_profile_views":{"id":"510","type":"ARTIST","area":"PHOTOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:48-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"EVENTS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:50-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:52-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"VIDEOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:55-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"HOME"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:32-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:43-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}

怎么了？

谢谢。

4

1 回答 1

2

我找到了答案。

出现此问题是因为配置 xpack.managed.* 使用集中式管道功能。

使用集中式管道，您无需创建任何 pipeline.yml 文件来配置本地 logstash 管道。使用集中式管道，您可以将自己的管道放入 elasticsearch。这是xpack的资源。

当您将 logstash 配置为由 elasticsearch 管理时，管道将通过 id 从 logstash 加载。

可以在此处找到有关集中式管道的更多信息：https ://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html

于 2018-07-16T21:05:20.920 回答