我有三个命名空间：使用 prometheus-operator 的命名空间监控、使用RabbitMq队列管理器和 prometheus-adapter 的命名空间 rabbitmq、使用仅为 RabbitMq pod 创建输入的应用程序的命名空间工作者。我想使用 Horizontal Pod Autoscaler (HPA)使用来自 RabbitMq pod（在rabbitmq命名空间上）的队列“task_queue”中的指标来扩展 worker pod（在worker命名空间上）。所有这些指标都由 prometheus 操作员（在监控命名空间上）收集，并显示在 prometheus 前端：
在 prometheus-url:8080/graph 查询“rabbitmq_queue_messages”：
rabbitmq_queue_messages{durable="true",endpoint="metrics",instance="x.x.x.x:9419",job="rabbitmq-server",namespace="rabbitmq",pod="rabbitmq-server-0",queue="task_queue",service="rabbitmq-server",vhost="/"}
RabbitMQ、Prometheus-operator 和 Prometheus-adapter 是从 helm 图表安装的
RabbitMQ（values.yaml 有密码并在 9419 启用指标以进行抓取）：
helm install --namespace rabbitmq rabbitmq-server stable/rabbitmq \
--set extraPlugins=rabbitmq_prometheus \
-f charts/default/rabbitmq/values.yaml
普罗米修斯适配器：
helm upgrade --install --namespace rabbitmq prometheus-adapter stable/prometheus-adapter \
--set prometheus.url="http://pmt-server-prometheus-oper-prometheus.monitoring.svc" \
--set prometheus.port="9090"
普罗米修斯操作员：
helm upgrade --install --namespace monitoring pmt-server stable/prometheus-operator \
--set prometheusOperator.createCustomResource=false \
-f charts/default/values.yaml
普罗米修斯值.yaml：
prometheus:
additionalServiceMonitors:
- name: rabbitmq-svc-monitor
selector:
matchLabels:
app: rabbitmq
namespaceSelector:
matchNames:
- rabbitmq
endpoints:
- port: metrics
interval: 10s
path: /metrics
自定义指标没问题：
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages"
},
"items": [
{
"describedObject": {
"kind": "Service",
"namespace": "rabbitmq",
"name": "rabbitmq-server",
"apiVersion": "/v1"
},
"metricName": "rabbitmq_queue_messages",
"timestamp": "2020-08-20T12:15:39Z",
"value": "0",
"selector": null
}
]
}
这是我的 hpa.yaml：
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: rabbitmq-queue-worker-hpa
namespace: worker
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: worker
minReplicas: 1
maxReplicas: 50
metrics:
- type: Object
object:
metric:
name: rabbitmq_queue_messages
describedObject:
apiVersion: "/v1"
kind: Service
name: rabbitmq-server.rabbitmq.svc.cluster.local
target:
type: Value
value: 100
但是 hpa 不能像 kubectl describe 显示的那样工作：
kubectl describe hpa/rabbitmq-queue-worker-hpa -n worker
Name: rabbitmq-queue-worker-hpa
Namespace: worker
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: rabbitmq-scaling-demo-app
meta.helm.sh/release-namespace: worker
CreationTimestamp: Thu, 20 Aug 2020 08:42:32 -0300
Reference: Deployment/worker
Metrics: ( current / target )
"rabbitmq_queue_messages" on Service/rabbitmq-server.rabbitmq.svc.cluster.local (target value): <unknown> / 100
Min replicas: 1
Max replicas: 50
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server could not find the metric rabbitmq_queue_messages for services rabbitmq-server.rabbitmq.svc.cluster.local
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 60m (x12 over 63m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get object metric value: unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Warning FailedGetObjectMetric 60m (x13 over 63m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Warning FailedGetObjectMetric 58m (x3 over 58m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get services.custom.metrics.k8s.io rabbitmq-server.rabbitmq.svc.cluster.local)
Warning FailedGetObjectMetric 2m59s (x218 over 57m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server could not find the metric rabbitmq_queue_messages for services rabbitmq-server.rabbitmq.svc.cluster.local
我相信 HPA 正试图在工作命名空间上找到 RabbitMq 服务，
Warning FailedGetObjectMetric 60m (x13 over 63m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
但该服务位于rabbitmq命名空间上。我尝试使用兔子的服务 FQDN (rabbitmq-server.rabbitmq.svc.cluster.local) 和服务名称 (rabbitmq-server)。我错过了什么？有没有办法让它工作？这里的重点是我有另一个项目有 10 多个命名空间，并且它们都使用相同的 rabbit 服务器（在rabbitmq命名空间上），所以让它们一起在同一个命名空间中将是一场噩梦。谢谢。
编辑 1：我的自定义指标config.yaml
prometheus:
url: http://pmt-server-prometheus-oper-prometheus.monitoring.svc
port: 9090
rbac:
create: true
serviceAccount:
create: true
service:
port: 443
logLevel: 6
rules:
custom:
- seriesQuery: 'rabbitmq_queue_messages{namespace!="",service!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
service: {resource: "service"}
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,queue="task_queue"}) by (<<.GroupBy>>)
适配器 helm 使用此文件安装：
helm upgrade --install --namespace rabbitmq prometheus-adapter stable/prometheus-adapter -f config.yaml
如果 HPA 是在 rabbitmq 命名空间中创建的，这是 HPA 描述：
Name: rabbitmq-queue-worker-hpa
Namespace: rabbitmq
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: rabbitmq-scaling-demo-app
meta.helm.sh/release-namespace: worker
CreationTimestamp: Fri, 21 Aug 2020 08:45:25 -0300
Reference: Deployment/worker
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 80%
Min replicas: 1
Max replicas: 50
Deployment pods: 0 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: deployments/scale.apps "worker" not found
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 9s (x17 over 4m11s) horizontal-pod-autoscaler deployments/scale.apps "worker" not found