I have been trying to use some custom, local built image on my kind cluster by following the instructions on https://kind.sigs.k8s.io/docs/user/local-registry - i.e. applying the following containerdConfigPatches
to my cluster.cfg
:
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.82:5000"]
endpoint = ["http://192.168.83.82:5000"]
192.168.83.82:5000
is the IP of the VM where the local (insecure) registry is running alongside the kind cluster and its exposed port.
Once the cluster is created, I can verify the settings where applied to all nodes:
docker exec kind-worker3 cat /etc/containerd/config.toml
# [...]
# Relevant sections:
# [...]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.82:5000"]
endpoint = ["http://192.168.83.82:5000"]
However, pods keep failing with to ErrImagePull
and these Event log:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32s default-scheduler Successfully assigned default/test-6bc95ff8c5-g6g86 to kind-worker3
Normal Pulled 31s kubelet, kind-worker3 Container image "docker.elastic.co/beats/filebeat-oss:6.4.2" already present on machine
Normal Created 31s kubelet, kind-worker3 Created container test-log-agent
Normal Started 31s kubelet, kind-worker3 Started container test-log-agent
Normal Pulling 16s (x2 over 31s) kubelet, kind-worker3 Pulling image "192.168.83.82:5000/test/image:2.2.1"
Warning Failed 16s (x2 over 31s) kubelet, kind-worker3 Failed to pull image "192.168.83.82:5000/test/image:2.2.1": rpc error: code = Unknown desc = failed to resolve image "192.168.83.82:5000/test/image:2.2.1": no available registry endpoint: failed to do request: Head https://192.168.83.82:5000/v2/test/image/manifests/2.2.1: http: server gave HTTP response to HTTPS client
Warning Failed 16s (x2 over 31s) kubelet, kind-worker3 Error: ErrImagePull
Normal BackOff 3s (x3 over 30s) kubelet, kind-worker3 Back-off pulling image "192.168.83.82:5000/test/image:2.2.1"
Warning Failed 3s (x3 over 30s) kubelet, kind-worker3 Error: ImagePullBackOff
TL;DR: "http: server gave HTTP response to HTTPS client" - which I thought would be solved with the ConfigPatch above (As it happens when you tweak docker's daemon.json
).
Also, as an alternative, tried loading the image from the host to the cluster nodes:
kind load docker-image 192.168.83.82:5000/test/image:2.2.1 --name="kind-cluster"
And verified the image was loaded to all nodes by listing them:
sysadmin@ubuntu:~/kind$ docker exec kind-worker3 crictl images
IMAGE TAG IMAGE ID SIZE
192.168.83.82:5000/test/image 2.2.1 ba1601dfa9c48 822MB
docker.io/kindest/kindnetd 0.5.0 ef97cccdfdb50 83.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180f 40.5MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4ff 258MB
k8s.gcr.io/kube-apiserver v1.15.3 be321f2ded3f3 249MB
k8s.gcr.io/kube-controller-manager v1.15.3 ac7d3fe5b34b7 200MB
k8s.gcr.io/kube-proxy v1.15.3 d428039608992 97.3MB
k8s.gcr.io/kube-scheduler v1.15.3 a44f53b10fee0 96.5MB
k8s.gcr.io/pause 3.1 da86e6ba6ca19 746kB
Hoping that the image now would be read from the image cache. However, the result was exactly the same.
Any clue on how to approach this one? Is there something I might have overlooked?