我正在尝试使用 Prometheus 自定义指标和Prometheus Adapter自动扩展 Kubernetes 部署和 HPA 。这些自定义指标通过另一个命名空间中的另一个部署发布到 Prometheus,每分钟查询一个 REST API 以获取特定指标,然后将该指标的值发布到 Prometheus。从那里,适配器应该能够向 Prometheus 查询所述指标,并使用一些附加标签作为查询条件,并使用新名称发布该指标。从那里,HPA 应该能够获取此指标并根据其值进行缩放。
这里是我的部署的标签,适配器应该基于它的匹配:
Labels: app.kubernetes.io/instance=event-subscription-dev-dev
app.kubernetes.io/managed-by=Tiller-dev
app.kubernetes.io/name=event-subscription-dev
deployment-name=event-subscription-webhook-worker-dev
helm.sh/chart=event-subscription-0.1.0-dev
以下是 Prometheus Adapter Helm 图表值/适配器规则:
logLevel: 1
metricsRelistInterval: 5s
prometheus:
url: 'http://<prometheus-url>'
rules:
custom:
- seriesQuery: '{__name__="event_subscription_current_message_lag"}'
name:
matches: "(.*)"
as: '${1}_webhooks'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
metricsQuery: 'sum(event_subscription_current_message_lag{queue="webhooks", container_name!="POD"})'
- seriesQuery: '{__name__="event_subscription_current_message_lag"}'
name:
matches: "(.*)"
as: '${1}_webhook_retries'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
metricsQuery: 'sum(event_subscription_current_message_lag{queue="webhook_retries", container_name!="POD"})'
这是metrics
我的 HPA 规范的一部分:
metrics:
- type: Pods
pods:
metric:
name: event_subscription_current_message_lag_webhooks
target:
type: AverageValue
averageValue: 10
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 85
我在这里遇到的问题不是适配器查询指标然后发布新指标,而是新指标没有与原始指标相关联的值。
例如,如果我运行,kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
我确实看到了我的event_subscription_current_message_lag_webhooks
和event_subscription_current_message_lag_webhook_retries
指标,但它们不像原始event_subscription_current_message_lag
指标那样具有任何价值。
这是来自的输出kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/*/event_subscription_current_message_lag"
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "event-subscription",
"name": "activemq-message-lag-retrieval-7bfc46b948-jr8kp",
"apiVersion": "/v1"
},
"metricName": "event_subscription_current_message_lag",
"timestamp": "2019-11-08T22:09:53Z",
"value": "1"
}
]
}
event_subscription_current_message_lag_webhooks
这是and的输出event_subscription_current_message_lag_webhook_retries
:
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag_webhooks"
},
"items": []
}
...
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/event-subscription/pods/%2A/event_subscription_current_message_lag_webhook_retries"
},
"items": []
}
我对适配器如何能够找到我的原始指标、查询它、发布新指标感到困惑,但如果没有值,我希望在这种情况下是1
.