Fixing 'cert-manager certificate expired' in Kubernetes
Comprehensive guide to diagnosing and fixing expired cert-manager certificates in Kubernetes. Learn how to force renewal, troubleshoot Issuer errors, and fix AC
- Certificates often expire due to failing ACME challenges (HTTP-01 or DNS-01) preventing auto-renewal.
- Check the `Certificate` resource status and `cert-manager` controller logs for explicit error messages like 'Failed to renew certificate'.
- Ensure your `ClusterIssuer` or `Issuer` is correctly configured and has valid credentials (e.g., DNS provider API keys).
- Manually trigger a renewal by deleting the underlying `Secret` or using the `cmctl renew` command.
| Method | When to Use | Time | Risk |
|---|---|---|---|
| cmctl renew | When the configuration is correct but a manual trigger is needed. | 1 min | Low |
| Delete Secret | When you want to force cert-manager to completely recreate the cert and secret. | 2 mins | Medium (brief TLS downtime if no valid cert exists) |
| Fix ACME Challenge | When Order/Challenge resources show 'pending' or 'invalid' status. | 10-30 mins | Low |
| Update Issuer Credentials | When DNS-01 provider API keys have expired or permissions changed. | 5 mins | Low |
Understanding the Error
When using cert-manager in Kubernetes, certificates are designed to automatically renew before they expire (typically 30 days before expiration for Let's Encrypt 90-day certs). If you encounter a cert-manager certificate expired situation, it means the automated renewal process failed silently or continuously failed until the validity period elapsed.
Common symptoms include:
- Browsers displaying
NET::ERR_CERT_DATE_INVALID. - Applications failing to communicate over mTLS.
kubectl get certificateshowingREADY=Falseor an oldNOT AFTERdate.
Step 1: Diagnose the Failure
The first step is to identify why the renewal failed. cert-manager creates several resources during the issuance process: Certificate -> CertificateRequest -> Order -> Challenge.
Start by inspecting the Certificate object:
kubectl describe certificate <cert-name> -n <namespace>
Look at the Events section. You might see errors like:
Issuing: certificate issuance in progress. This may take some time.(Stuck in issuing)Re-queuing item due to error processing: ...
Next, check the CertificateRequest and Order:
kubectl get certificaterequest,order -n <namespace>
kubectl describe order <order-name> -n <namespace>
If the Order is stuck, inspect the Challenge:
kubectl get challenge -n <namespace>
kubectl describe challenge <challenge-name> -n <namespace>
The Challenge will usually tell you exactly what went wrong. For example, for HTTP-01, it might fail to reach the .well-known/acme-challenge endpoint due to an Ingress misconfiguration. For DNS-01, it might fail to propagate the TXT record.
Step 2: Common Root Causes and Fixes
1. HTTP-01 Challenge Failures
If using HTTP-01, Let's Encrypt must be able to reach your cluster over port 80.
- Check Ingress: Ensure your Ingress controller is routing traffic correctly. cert-manager creates a temporary solver Pod and Ingress. If your default Ingress class changed or is misconfigured, the challenge will fail.
- Check Firewalls: Ensure port 80 is open to the internet.
2. DNS-01 Challenge Failures
If using DNS-01, cert-manager needs to create a TXT record in your DNS provider.
- Expired Credentials: The API token or IAM role used by the
ClusterIssuerto authenticate with Route53, Cloudflare, etc., might have expired or lack permissions. - Check Issuer Status:
Look forkubectl describe clusterissuer <issuer-name>Status: Trueand typeReady.
3. cert-manager Controller Issues
Sometimes the cert-manager controller itself is crash-looping or stuck.
- Check Controller Logs:
Look for explicit errors or rate-limiting messages from Let's Encrypt (kubectl logs -n cert-manager -l app=cert-manager -f429 Too Many Requests).
Step 3: Forcing a Renewal
Once you have resolved the underlying issue (e.g., fixed the Ingress route or updated the DNS API key), you need to tell cert-manager to try again.
Option A: Using cmctl (Recommended) The official CLI tool is the safest way to request renewal:
cmctl renew <cert-name> -n <namespace>
Option B: Deleting the Secret
If you don't have cmctl, you can delete the Kubernetes Secret that holds the expired certificate. This forces cert-manager to issue a new one immediately:
kubectl delete secret <secret-name> -n <namespace>
Note: This will cause a brief period where no certificate is served until the new one is provisioned.
Frequently Asked Questions
# 1. Check the status of the Certificate
kubectl get certificate -A
# 2. Describe the failing Certificate to see events
kubectl describe certificate my-app-cert -n production
# 3. Check for stuck Orders and Challenges
kubectl get order,challenge -n production
kubectl describe challenge <challenge-name> -n production
# 4. Check cert-manager controller logs for systemic issues or Let's Encrypt rate limits
kubectl logs -n cert-manager deploy/cert-manager
# 5. Force renewal using cmctl once the root cause (e.g., DNS/Ingress) is fixed
cmctl renew my-app-cert -n production
# Alternative: Force renewal by deleting the secret
# kubectl delete secret my-app-tls-secret -n productionError Medic Editorial
Error Medic Editorial is composed of senior SREs and DevOps practitioners dedicated to solving complex infrastructure challenges. With decades of combined experience in Kubernetes, cloud-native technologies, and incident response, our guides provide battle-tested solutions for production systems.