My goal is do deploy a self-hosted instance of GitLab on EKS. I have read through the guide on the GitLab docs and am attempting the operator install method. I setup my cluster using eksctl v0.61.0 and three t4g.large instances. The cluster comes up and seems healthy.
kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/aws-node-9k7mg 1/1 Running 0 3m25s
kube-system pod/aws-node-hlkxr 1/1 Running 0 3m25s
kube-system pod/aws-node-rc5br 1/1 Running 0 3m24s
kube-system pod/coredns-5c778788f4-cw5gq 1/1 Running 0 15m
kube-system pod/coredns-5c778788f4-ff8mn 1/1 Running 0 15m
kube-system pod/kube-proxy-hrxtz 1/1 Running 0 3m25s
kube-system pod/kube-proxy-phw7p 1/1 Running 0 3m25s
kube-system pod/kube-proxy-rtlgj 1/1 Running 0 3m25s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 16m
kube-system service/kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP 16m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-node 3 3 3 3 3 <none> 16m
kube-system daemonset.apps/kube-proxy 3 3 3 3 3 <none> 16m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 2/2 2 2 16m
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-5c778788f4 2 2 2 15m
I begin by installing cert-manager v1.6.0 using the default config.
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.0/cert-manager.yaml
kubectl get all -n cert-manager
NAME READY STATUS RESTARTS AGE
pod/cert-manager-77fd97f598-wxtj8 1/1 Running 0 18s
pod/cert-manager-cainjector-7974c84449-ghlfr 1/1 Running 0 18s
pod/cert-manager-webhook-5f4b965fbd-8kqv2 1/1 Running 0 17s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.100.71.170 <none> 9402/TCP 18s
service/cert-manager-webhook ClusterIP 10.100.191.224 <none> 443/TCP 18s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 19s
deployment.apps/cert-manager-cainjector 1/1 1 1 19s
deployment.apps/cert-manager-webhook 1/1 1 1 18s
NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-77fd97f598 1 1 1 19s
replicaset.apps/cert-manager-cainjector-7974c84449 1 1 1 19s
replicaset.apps/cert-manager-webhook-5f4b965fbd 1 1 1 18s
Next, I install the metrics server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
This comes up as well and does not appear to have any issues.
Finally, I attempt to install the GitLab operator
GL_OPERATOR_VERSION=0.1.0
PLATFORM=kubernetes
kubectl create namespace gitlab-system
kubectl apply -f https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${GL_OPERATOR_VERSION}/gitlab-operator-${PLATFORM}-${GL_OPERATOR_VERSION}.yaml
*Note: The newest release of cert-manager at the time of this post was 1.6.0. During this update, the APIVersions v1alpha2, v1alpha3, and v1beta1 were deprecated. When I attempted this install it failed to create the Issuer and Certificate. Updating the APIVersions to cert-manager.io/v1 fixed this.
Now, it creates all the resources.
kubectl get all -n gitlab-system
NAME READY STATUS RESTARTS AGE
pod/gitlab-controller-manager-ccd797cb6-9c428 0/2 CrashLoopBackOff 4 30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/gitlab-controller-manager-metrics-service ClusterIP 10.100.252.76 <none> 8443/TCP 30s
service/gitlab-webhook-service ClusterIP 10.100.85.217 <none> 443/TCP 30s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/gitlab-controller-manager 0/1 1 0 30s
NAME DESIRED CURRENT READY AGE
replicaset.apps/gitlab-controller-manager-ccd797cb6 1 1 0 30s
As shown, the pod/gitlab-controller-manager-ccd797cb6-9c428 is in a CrashLoopBackOff state. It continues to restart indefinitely.
kubectl describe pod gitlab-controller-manager-ccd797cb6-9c428 -n gitlab-system
Name: gitlab-controller-manager-ccd797cb6-9c428
Namespace: gitlab-system
Priority: 0
Node: ip-192-168-78-2.us-east-2.compute.internal/192.168.78.2
Start Time: Thu, 28 Oct 2021 18:13:28 -0400
Labels: control-plane=controller-manager
pod-template-hash=ccd797cb6
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 192.168.95.73
IPs:
IP: 192.168.95.73
Controlled By: ReplicaSet/gitlab-controller-manager-ccd797cb6
Containers:
manager:
Container ID: docker://8576f635b72389a824284a1c342c390036af50bf85a60aa3299af17d77764971
Image: registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator@sha256:3d0ff0fc176511d67f3784060023157fbdaed8109539f3d340d68ac8f18d6425
Ports: 9443/TCP, 6060/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/manager
Args:
--metrics-addr=127.0.0.1:8080
--enable-leader-election
--zap-devel=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 28 Oct 2021 18:14:24 -0400
Finished: Thu, 28 Oct 2021 18:14:24 -0400
Ready: False
Restart Count: 3
Limits:
cpu: 200m
memory: 300Mi
Requests:
cpu: 200m
memory: 100Mi
Liveness: http-get http://:health-port/liveness delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:health-port/readiness delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
WATCH_NAMESPACE: gitlab-system (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from gitlab-manager-token-vjdfx (ro)
kube-rbac-proxy:
Container ID: docker://1db8028b18e0e7f255f1fdc1c0ab086d0cb01d17a10e3b0d17b9a8e6afda9175
Image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
Image ID: docker-pullable://gcr.io/kubebuilder/kube-rbac-proxy@sha256:e10d1d982dd653db74ca87a1d1ad017bc5ef1aeb651bdea089debf16485b080b
Port: 8443/TCP
Host Port: 0/TCP
Args:
--secure-listen-address=0.0.0.0:8443
--upstream=http://127.0.0.1:8080/
--logtostderr=true
--v=10
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 28 Oct 2021 18:14:24 -0400
Finished: Thu, 28 Oct 2021 18:14:24 -0400
Ready: False
Restart Count: 3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from gitlab-manager-token-vjdfx (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-server-cert
Optional: false
gitlab-manager-token-vjdfx:
Type: Secret (a volume populated by a Secret)
SecretName: gitlab-manager-token-vjdfx
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 61s default-scheduler Successfully assigned gitlab-system/gitlab-controller-manager-ccd797cb6-9c428 to ip-192-168-78-2.us-east-2.compute.internal
Warning FailedMount 60s (x2 over 61s) kubelet MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found
Normal Pulling 55s kubelet Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
Normal Pulled 55s kubelet Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 3.560963186s
Normal Pulled 53s kubelet Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" in 1.650875485s
Normal Pulled 52s kubelet Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 52s (x2 over 53s) kubelet Created container kube-rbac-proxy
Normal Started 52s (x2 over 53s) kubelet Started container kube-rbac-proxy
Normal Pulled 52s kubelet Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 490.074654ms
Warning BackOff 50s (x2 over 51s) kubelet Back-off restarting failed container
Warning BackOff 50s (x2 over 51s) kubelet Back-off restarting failed container
Normal Pulling 39s (x3 over 59s) kubelet Pulling image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0"
Normal Started 38s (x3 over 55s) kubelet Started container manager
Normal Created 38s (x3 over 55s) kubelet Created container manager
Normal Pulled 38s kubelet Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 512.734325ms
The only issue I recognize is the missing "webhook-server-cert" secret.
kubectl get secrets -n gitlab-system
NAME TYPE DATA AGE
default-token-tzxs2 kubernetes.io/service-account-token 3 86s
gitlab-app-token-7btgp kubernetes.io/service-account-token 3 83s
gitlab-manager-token-vjdfx kubernetes.io/service-account-token 3 83s
gitlab-nginx-ingress-token-v5jdh kubernetes.io/service-account-token 3 82s
webhook-server-cert kubernetes.io/tls 3 80s
The secret is there and when I run get on it I can see the cert and key.
Here is the result of running kubectl get events -n gitlab-system
LAST SEEN TYPE REASON OBJECT MESSAGE
100s Normal Scheduled pod/gitlab-controller-manager-ccd797cb6-9c428 Successfully assigned gitlab-system/gitlab-controller-manager-ccd797cb6-9c428 to ip-192-168-78-2.us-east-2.compute.internal
99s Warning FailedMount pod/gitlab-controller-manager-ccd797cb6-9c428 MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found
78s Normal Pulling pod/gitlab-controller-manager-ccd797cb6-9c428 Pulling image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0"
94s Normal Pulled pod/gitlab-controller-manager-ccd797cb6-9c428 Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 3.560963186s
77s Normal Created pod/gitlab-controller-manager-ccd797cb6-9c428 Created container manager
77s Normal Started pod/gitlab-controller-manager-ccd797cb6-9c428 Started container manager
94s Normal Pulling pod/gitlab-controller-manager-ccd797cb6-9c428 Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
92s Normal Pulled pod/gitlab-controller-manager-ccd797cb6-9c428 Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" in 1.650875485s
91s Normal Created pod/gitlab-controller-manager-ccd797cb6-9c428 Created container kube-rbac-proxy
91s Normal Started pod/gitlab-controller-manager-ccd797cb6-9c428 Started container kube-rbac-proxy
91s Normal Pulled pod/gitlab-controller-manager-ccd797cb6-9c428 Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 490.074654ms
91s Normal Pulled pod/gitlab-controller-manager-ccd797cb6-9c428 Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" already present on machine
89s Warning BackOff pod/gitlab-controller-manager-ccd797cb6-9c428 Back-off restarting failed container
89s Warning BackOff pod/gitlab-controller-manager-ccd797cb6-9c428 Back-off restarting failed container
77s Normal Pulled pod/gitlab-controller-manager-ccd797cb6-9c428 Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator:0.1.0" in 512.734325ms
100s Normal SuccessfulCreate replicaset/gitlab-controller-manager-ccd797cb6 Created pod: gitlab-controller-manager-ccd797cb6-9c428
100s Normal ScalingReplicaSet deployment/gitlab-controller-manager Scaled up replica set gitlab-controller-manager-ccd797cb6 to 1
99s Normal cert-manager.io certificaterequest/gitlab-serving-cert-ghlz8 Certificate request has been approved by cert-manager.io
99s Warning BadConfig certificaterequest/gitlab-serving-cert-ghlz8 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
99s Normal CertificateIssued certificaterequest/gitlab-serving-cert-ghlz8 Certificate fetched from issuer successfully
99s Normal Issuing certificate/gitlab-serving-cert Issuing certificate as Secret does not exist
99s Normal Generated certificate/gitlab-serving-cert Stored new private key in temporary Secret resource "gitlab-serving-cert-k5djd"
99s Normal Requested certificate/gitlab-serving-cert Created new CertificateRequest resource "gitlab-serving-cert-ghlz8"
99s Normal Issuing certificate/gitlab-serving-cert The certificate has been successfully issued
I'm not sure how to approach this issue. Any insights?