We are using Cert-manager to manage the tls certifications for a website. The website's certificate expired yesterday, I tried to investigate why cert-manager was not doing its job.
I have checked the details of certificate fakename-io-cert, looks like cert-manager tried to renew the cert one month ago?:
$ kubectl describe cert/fakename-io-cert
Name: fakename-io-cert
Namespace: stage
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2020-10-28T17:10:08Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:commonName:
f:dnsNames:
f:issuerRef:
.:
f:kind:
f:name:
f:privateKey:
.:
f:rotationPolicy:
f:secretName:
Manager: kubectl
Operation: Update
Time: 2020-10-28T17:10:08Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:nextPrivateKeySecretName:
f:notAfter:
f:notBefore:
f:renewalTime:
f:revision:
Manager: controller
Operation: Update
Time: 2021-01-27T20:26:11Z
Resource Version: 28153132
Self Link: /apis/cert-manager.io/v1/namespaces/stage/certificates/fakename-io-cert
UID: 193717dd-0c00-43c5-8bde-5b7f981a5558
Spec:
Common Name: *.fakename.io
Dns Names:
fakename.io
*.fakename.io
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Private Key:
Rotation Policy: Always
Secret Name: cert-stage-wildcard
Status:
Conditions:
Last Transition Time: 2021-01-27T20:26:11Z
Message: Certificate expired on Tue, 26 Jan 2021 16:10:10 UTC
Reason: Expired
Status: False
Type: Ready
Last Transition Time: 2020-12-27T16:10:10Z
Message: Renewing certificate as renewal was scheduled at 2020-12-27 16:10:10 +0000 UTC
Reason: Renewing
Status: True
Type: Issuing
Next Private Key Secret Name: fakename-io-cert-cjmpk
Not After: 2021-01-26T16:10:10Z
Not Before: 2020-10-28T16:10:10Z
Renewal Time: 2020-12-27T16:10:10Z
Revision: 1
Events: <none>
The following are the information of other related resources, such as certificaterequests, certificates, secrets:
$ kubectl get certificaterequests
NAME READY AGE
fakename-io-cert-8nxb6 False 31d
fakename-io-cert-k79kq True 91d
$ kubectl get certificates
NAME READY SECRET AGE
fakename-io-cert False cert-stage-wildcard 91d
$ kubectl get secrets
NAME TYPE DATA AGE
cert-stage-wildcard kubernetes.io/tls 2 91d
fakename-io-cert-cjmpk Opaque 1 31d
$ kubectl describe secrets/cert-stage-wildcard
Name: cert-stage-wildcard
Namespace: stage
Labels: <none>
Annotations: cert-manager.io/alt-names: *.fakename.io,fakename.io
cert-manager.io/certificate-name: fakename-io-cert
cert-manager.io/common-name: *.fakename.io
cert-manager.io/ip-sans:
cert-manager.io/issuer-group:
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-prod
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 3570 bytes
tls.key: 1675 bytes
$ kubectl describe secrets/fakename-io-cert-cjmpk
Name: fakename-io-cert-cjmpk
Namespace: stage
Labels: cert-manager.io/next-private-key=true
Annotations: <none>
Type: Opaque
Data
====
tls.key: 1700 bytes
And then the ClusterIssuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
email: someone@fakename.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: issuer-account-key
solvers:
- selector:
dnsNames:
- fakename.io
dns01:
digitalocean:
tokenSecretRef:
name: digitalocean-dns
key: do-access-token
Anyone has any idea of why this is not working?
UPDATE:
The following logs are found in cert-manager's logs:
I0122 20:40:15.494843 1 reflector.go:207] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0122 20:40:15.495228 1 reflector.go:207] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
E0122 20:40:15.527258 1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"fakename-io-cert-8nxb6\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="fakename-io-cert-8nxb6" "related_resource_namespace"="stage" "resource_kind"="Order" "resource_name"="fakename-io-cert-8nxb6-31268985" "resource_namespace"="stage" "resource_version"="v1"
E0122 20:40:15.540473 1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"fakename-io-cert-k79kq\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="fakename-io-cert-k79kq" "related_resource_namespace"="stage" "resource_kind"="Order" "resource_name"="fakename-io-cert-k79kq-31268985" "resource_namespace"="stage" "resource_version"="v1"
E0122 20:40:17.801204 1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="stage/fakename-io-cert-k79kq-31268985"
E0122 20:40:17.850290 1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="stage/fakename-io-cert-8nxb6-31268985"
I0122 20:40:17.917857 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0122 20:40:17.919532 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
E0122 20:40:23.180388 1 sync.go:110] cert-manager/controller/orders "msg"="Failed to determine the list of Challenge resources needed for the Order" "error"="no configured challenge solvers can be used for this challenge" "resource_kind"="Order" "resource_name"="fakename-io-cert-8nxb6-31268985" "resource_namespace"="stage" "resource_version"="v1"
I0122 20:40:23.490123 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0122 20:40:23.502923 1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"