SSL Certificates (Let's Encrypt + cert-manager)

Cairn uses cert-manager with Let's Encrypt to automatically provision and renew TLS certificates for all ingress endpoints.


Architecture

Internet ──► Ingress-NGINX ──► cert-manager (HTTP-01 challenge)
                │
                ├── https://platform.dev
                ├── https://api.platform.dev
                ├── https://argocd.platform.dev
                ├── https://grafana.platform.dev
                ├── https://portainer.platform.dev
                └── https://minio.platform.dev

All subdomains share a single wildcard-ish certificate stored in the platform-tls secret.


1. Install cert-manager via Helm

Add Helm Repository

helm repo add cert-manager https://charts.jetstack.io
helm repo update

Install with CRDs

helm upgrade --install cert-manager cert-manager/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true \
  --wait

Verify Installation

kubectl get pods -n cert-manager
# NAME                                       READY   STATUS    RESTARTS
# cert-manager-7dd585dc66-abcde              1/1     Running   0
# cert-manager-cainjector-7dd585dc66-abcde   1/1     Running   0
# cert-manager-webhook-7dd585dc66-abcde      1/1     Running   0

kubectl get crd | grep cert-manager
# certificaterequests.cert-manager.io
# certificates.cert-manager.io
# challenges.acme.cert-manager.io
# clusterissuers.cert-manager.io
# issuers.cert-manager.io
# orders.acme.cert-manager.io

2. Create ClusterIssuer (Staging + Production)

Staging Issuer (for testing)

The staging environment has much higher rate limits but issues invalid certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: nginx

Production Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Apply

kubectl apply -f clusterissuer.yaml

Verify

kubectl get clusterissuer
# NAME                  READY   AGE
# letsencrypt-prod      True    1m
# letsencrypt-staging   True    1m

kubectl describe clusterissuer letsencrypt-prod
# Should show: Status: True  Message: "The ACME account was registered with the ACME server"

3. Annotate Ingress for Automatic Certificate Provisioning

The Cairn Ingress is annotated to automatically request certificates from cert-manager:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: platform
  namespace: platform
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod    # <-- triggers cert-manager
spec:
  tls:
  - hosts:
    - platform.dev
    - api.platform.dev
    - argocd.platform.dev
    - grafana.platform.dev
    - portainer.platform.dev
    - minio.platform.dev
    secretName: platform-tls    # <-- cert stored here
  rules:
  - host: platform.dev
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: platform-portal
            port:
              number: 80

Key Annotations

Annotation Value Purpose
cert-manager.io/cluster-issuer letsencrypt-prod Issuer to use for certificate
kubernetes.io/ingress.class nginx Which ingress controller to use
nginx.ingress.kubernetes.io/proxy-body-size 50m Allow larger request bodies

Test with Staging First

To test without hitting production rate limits, temporarily switch:

kubectl annotate ingress platform -n platform \
  cert-manager.io/cluster-issuer=letsencrypt-staging \
  --overwrite

After verification, switch back:

kubectl annotate ingress platform -n platform \
  cert-manager.io/cluster-issuer=letsencrypt-prod \
  --overwrite

4. Certificate Status & Renewal Monitoring

Check Certificate

kubectl get certificate -n platform
# NAME           READY   SECRET         AGE
# platform-tls   True    platform-tls   10d

Detailed Certificate Info

kubectl describe certificate platform-tls -n platform

Look for:

Status:
  Conditions:
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2026-09-28T12:00:00Z
  Renewal Time:            2026-08-29T12:00:00Z

Check the TLS Secret

kubectl get secret platform-tls -n platform -o yaml

Check Certificate Order

kubectl get orders -n platform
kubectl describe order -n platform <order-name>

Check ACME Challenges

kubectl get challenges -n platform
kubectl describe challenge -n platform <challenge-name>

Set Up Renewal Alerts

cert-manager handles automatic renewal 30 days before expiry. To monitor proactively:

# Check days until expiry
kubectl get certificate platform-tls -n platform -o json | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status']['notAfter'])"

# Add a Prometheus alert for certificates expiring soon
# cert-manager exposes metrics at :9402/metrics
# certmanager_certificate_expiration_timestamp_seconds

5. Troubleshooting DNS Challenges

Symptom: Certificate stuck at "Issuing"

kubectl describe certificate platform-tls -n platform

Check the order and challenge:

kubectl get orders -n platform
kubectl describe order -n platform <order-name>
# Look for: State: pending, valid, or failed

Symptom: Challenge "pending" or "invalid"

kubectl get challenges -n platform
kubectl describe challenge -n platform <challenge-name>

Common DNS Issues

Error Cause Fix
Failed to load http-01 solver Ingress not reachable from internet Ensure port 80 is open; test with curl http://DOMAIN/.well-known/acme-challenge/test
No valid IP addresses DNS not resolving to server IP dig +short DOMAIN, verify A record
404 on challenge URL Ingress not routing to cert-manager solver Check ingress controller is running: kubectl get pods -n ingress-nginx
Rate limit exceeded Too many certificates in a week Use staging issuer for testing
context deadline exceeded Network timeout from ACME server Check egress firewall; Let's Encrypt must reach your server
Unknown issuer ClusterIssuer name mismatch Verify annotation: kubectl get ingress platform -n platform -o yaml | grep cluster-issuer

Manual Challenge Debugging

# 1. Verify DNS resolution
dig +short platform.dev
dig +short argocd.platform.dev

# 2. Check ingress controller serves on port 80
kubectl get svc -n ingress-nginx ingress-nginx-controller

# 3. Test HTTP challenge endpoint from external
curl -v http://platform.dev/.well-known/acme-challenge/ 2>&1 | head -20

# 4. Check cert-manager pod logs
kubectl logs -n cert-manager deploy/cert-manager --tail=50
kubectl logs -n cert-manager deploy/cert-manager --tail=50 -f

# 5. Force re-issuance by deleting the certificate
kubectl delete certificate platform-tls -n platform
# Wait for cert-manager to recreate it from the ingress annotation

# 6. Delete the order to retry
kubectl delete orders -n platform --all

6. Manual Certificate (Non-Let's Encrypt)

If you have your own certificate:

kubectl create secret tls platform-tls \
  --namespace platform \
  --cert=/path/to/cert.pem \
  --key=/path/to/key.pem

# Remove the cert-manager annotation to prevent overwrite
kubectl annotate ingress platform -n platform \
  cert-manager.io/cluster-issuer- \

7. Let's Encrypt Rate Limits

Limit Scope Value
Certificates per registered domain Per week 50
Duplicate certificate Per week 5
Failed validations Per account, per hostname, per hour 5
ACME account registrations Per IP, per 3 hours 10

Always use staging for testing, especially during development.


8. Full Verification Checklist

# 1. cert-manager pods running
kubectl get pods -n cert-manager

# 2. ClusterIssuer ready
kubectl get clusterissuer letsencrypt-prod -o jsonpath='{.status.conditions[0].status}'

# 3. Ingress has correct annotation
kubectl get ingress platform -n platform -o jsonpath='{.metadata.annotations}'

# 4. TLS section lists all hosts
kubectl get ingress platform -n platform -o jsonpath='{.spec.tls}'

# 5. Certificate issued
kubectl get certificate platform-tls -n platform -o wide

# 6. TLS secret exists
kubectl get secret platform-tls -n platform

# 7. HTTPS works from outside
curl -I https://$DOMAIN 2>/dev/null | grep -i "SSL\|TLS\|certificate"
curl -v https://$DOMAIN 2>&1 | grep "Server certificate"