我正在使用 chaostoolkit，并且能够从命令行成功运行混沌实验。但是，当我尝试在 k8s 中运行与作业相同的作业时，它会抛出“连接被拒绝”错误。我觉得奇怪的是，有时稳态假设步骤成功运行并在终止 pod 操作失败时返回 200 OK，但很多时候它也在假设步骤本身失败（在终止 pod 的操作之前） . 顺便说一句，我正在谷歌云中这样做。
在某些运行期间，我看到动作之前的假设以及终止 pod 是成功的，但是动作之后的假设（终止）得到“连接被拒绝”错误。
任何帮助/提示表示赞赏。
这是错误消息：
[2022-02-03 07:24:54 DEBUG] [caching:35] Cached 2 activities
[2022-02-03 07:24:54 INFO] [experiment:54] Validating the experiment's syntax
[2022-02-03 07:24:54 DEBUG] [configuration:47] Loading configuration...
[2022-02-03 07:24:54 DEBUG] [secret:74] Loading secrets...
[2022-02-03 07:24:54 DEBUG] [secret:89] Secrets loaded
[2022-02-03 07:25:12 INFO] [experiment:103] Experiment looks valid
[2022-02-03 07:25:12 DEBUG] [caching:42] Clearing activities cache
[2022-02-03 07:25:12 DEBUG] [caching:25] Building activity cache...
[2022-02-03 07:25:12 DEBUG] [caching:35] Cached 2 activities
[2022-02-03 07:25:12 INFO] [experiment:182] Running experiment: What happens if we terminate an instance of the application?
[2022-02-03 07:25:12 DEBUG] [configuration:47] Loading configuration...
[2022-02-03 07:25:12 DEBUG] [secret:74] Loading secrets...
[2022-02-03 07:25:12 DEBUG] [secret:89] Secrets loaded
[2022-02-03 07:25:12 DEBUG] [__init__:39] Initializing controls
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'experiment'
[2022-02-03 07:25:12 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 07:25:12 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 07:25:12 DEBUG] [activity:233] Activity failed
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
这是我提供给工作的配置：
health-http.yaml: |
version: 1.0.0
title: What happens if we terminate an instance of the application?
description: If an instance of the application is terminated, the applications as a whole should still be operational.
tags:
- k8s
- pod
steady-state-hypothesis:
title: The app is healthy
probes:
- name: app-responds-to-requests
type: probe
tolerance: 200
provider:
type: http
timeout: 10
verify_tls: false
url: http://newapp
headers:
Host: newapp.example.com
method:
- type: action
name: terminate-app-pod
provider:
type: python
module: chaosk8s.pod.actions
func: terminate_pods
arguments:
label_selector: app=newapp
rand: true
ns: default
pauses:
after: 2
我能够 ssh 进入一个虚拟的 nginx pod 和“curl newapp”，它会返回正确的响应，所以该服务肯定是活跃的并且可以工作。除了其他权限外，我还创建了具有获取、列出、删除 Pod 的权限的服务帐户。
这是实验清单：
apiVersion: batch/v1
kind: Job
metadata:
name: newapp-chaos
spec:
activeDeadlineSeconds: 600
backoffLimit: 0
template:
metadata:
labels:
app: newapp
annotations:
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: newapp-chaos
restartPolicy: Never
containers:
- name: chaostoolkit
image: vfarcic/chaostoolkit:1.4.1-2
args:
- --verbose
- run
- /experiment/health-http.yaml
env:
- name: CHAOSTOOLKIT_IN_POD
value: "true"
volumeMounts:
- name: config
mountPath: /experiment
readOnly: true
resources:
limits:
cpu: 20m
memory: 64Mi
requests:
cpu: 20m
memory: 64Mi
volumes:
- name: config
configMap:
name: newapp-config
这是我的应用清单：
apiVersion: apps/v1
kind: Deployment
metadata:
name: newapp-v2
spec:
replicas: 1
selector:
matchLabels:
app: newapp
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: newapp
version: v2
spec:
containers:
- image: rstarmer/hostname:v2
imagePullPolicy: Always
name: newapp
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
labels:
app: newapp
name: newapp
spec:
#externalTrafficPolicy: Cluster
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: newapp
sessionAffinity: None
这是终止也很好但后来遇到错误的输出：
[2022-02-03 09:43:22 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 09:43:22 DEBUG] [activity:179] => succeeded with '{'status': 200, 'headers': {'Server': 'nginx/1.15.4', 'Date': 'Thu, 03 Feb 2022 09:43:22 GMT', 'Content-Type': 'text/html', 'Content-Length': '208', 'Last-Modified': 'Thu, 03 Feb 2022 07:21:47 GMT', 'Connection': 'keep-alive', 'ETag': '"61fb828b-d0"', 'Accept-Ranges': 'bytes'}, 'body': "<HTML>\n<HEAD>\n<TITLE>This page is on newapp-v2-866f8798cd-8s424 and is version v2</TITLE>\n</HEAD><BODY>\n<H1>THIS IS HOST newapp-v2-866f8798cd-8s424</H1>\n<H2>And we're running version: v2</H2>\n</BODY>\n</HTML>\n"}'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 DEBUG] [hypothesis:212] allowed tolerance is 200
[2022-02-03 09:43:22 INFO] [hypothesis:222] Steady state hypothesis is met!
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'method'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 INFO] [activity:160] Action: terminate-app-pod
[2022-02-03 09:43:22 DEBUG] [python:34] Activity 'terminate-app-pod' loaded from '/usr/local/lib/python3.8/site-packages/chaosk8s/pod/actions.py'
[2022-02-03 09:43:23 DEBUG] [actions:193] Found 3 pods labelled 'app=newapp' in ns default
[2022-02-03 09:43:23 DEBUG] [activity:181] => succeeded without any result value
[2022-02-03 09:43:23 INFO] [activity:197] Pausing after activity for 2s...
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'method'
[2022-02-03 09:43:25 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:25 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 09:43:25 DEBUG] [activity:233] Activity failed
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused