Podman, Caddy, and Lego: DNS-01 Challenges without having to use xCaddy
Takes only a few moments and ensures you don't have to rebuild with the right DNS plugins
Caddy is great and one of the main benefits it has is the automatic SSL. Unfortunately, that fails when I needed DNS challenges. The common solution is to rebuild Caddy with the proper plugins but I didn’t like this as it meant I had to watch out for updates. There are also unofficial builds but I don’t want to risk that.
This got me exploring other ways to get the certificates through Let’s Encrypt. I scrolled through their ACME clients and got into Lego.
Quadlets
I use Podman Quadlets and my existing Caddy is in a pod. The architecture is that it has its own Tailscale container and I connect to the container using Tailscale.
Ensure first that all folders are available (otherwise you will encounter errors).
mkdir -p ~/containers/caddy/lego ~/containers/caddy/state
touch ~/containers/caddy/Caddyfile
The command above creates the prerequisite files and folders for the below to work.
# ~/.config/containers/systemd/caddy/caddy.pod
[Unit]
Description=Caddy Pod
[Pod]
PodmanArgs=--hostname=tailscale-containers
AddHost=host.containers.internal:host-gateway
[Install]
WantedBy=default.target
The above is the pod which everyone connects to (except for Lego - more on that later). The app and the Tailscale will be in the same network since they’re in the same pod.
# ~/.config/containers/systemd/caddy/caddy-app.container
[Unit]
Description=Caddy App
[Container]
Pod=caddy.pod
Image=docker.io/caddy:alpine
AutoUpdate=registry
Volume=%h/containers/caddy/Caddyfile:/etc/caddy/Caddyfile:ro,Z
Volume=%h/containers/caddy/lego:/certs:ro,z
HealthCmd=curl -f http://localhost:2019/config/ || exit 1
HealthInterval=30s
HealthTimeout=5s
HealthRetries=3
HealthStartPeriod=10s
[Service]
Restart=always
Above is the actual Caddy app. What’s noted is that Volume=%h/containers/caddy/lego:/certs:ro,z MUST have z (lowercase) because it is shared with another container.
# ~/.config/containers/systemd/caddy/caddy-tailscale.container
[Unit]
Description=Tailscale
[Container]
Pod=caddy.pod
Image=docker.io/tailscale/tailscale:stable
AutoUpdate=registry
Volume=%h/containers/caddy/state:/var/lib/tailscale:Z
AddCapability=net_admin
AddDevice=/dev/net/tun:/dev/net/tun
Environment=TS_EXTRA_ARGS=--advertise-tags=tag:container
Environment=TS_STATE_DIR=/var/lib/tailscale
Environment=TS_USERSPACE=false
Environment=TS_ENABLE_HEALTH_CHECK=true
Environment=TS_LOCAL_ADDR_PORT=127.0.0.1:4000
Secret=ts-container,type=env,target=TS_AUTHKEY
HealthCmd=wget --no-verbose --tries=1 --spider http://127.0.0.1:4000/healthz || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=5s
HealthTimeout=3s
[Service]
Restart=always
Above is Tailscale which contains nothing special.
# ~/.config/containers/systemd/caddy/caddy-lego.container
[Unit]
Description=Lego
[Container]
Image=docker.io/goacme/lego:v5
AutoUpdate=registry
Secret=cf_api_token,type=env,target=CLOUDFLARE_DNS_API_TOKEN
Exec=run --accept-tos \
--dns cloudflare \
--path /certificates \
--dns.resolvers 1.1.1.1:53 \
--dns.resolvers 2606:4700:4700::1111:53 \
-d '*.pdlozano.com' \
-d 'pdlozano.com'
Volume=%h/containers/caddy/lego:/certificates:z
[Service]
Type=oneshot
ExecStartPost=podman exec systemd-caddy-app caddy reload --config=/etc/caddy/Caddyfile
Then there is Lego which is NOT connected to the Pod. This is a service purely to get certificates and update them. It does not communicate at all with Caddy or Tailscale so there is no need to expose this to the same pod.
Also note that everytime the Lego container runs, it reloads Caddy. This means no downtime at all if the certificates are new.
Note that if you follow my file naming conventions in the comment, the name for the container should be systemd-caddy-app. If you have another name, you should double check to ensure it works.
Systemd Timer
Caddy normally handles the renewal of certificates. For lego though, we must set it to renew it. I added the following Systemd timer below to do so. It runs daily at 0100H. While it does run daily, it only renews when 2/3 of the certificate lifetime is complete so there’s no need to worry about rate limits in Let’s Encrypt.
# ~/.config/systemd/user/caddy-lego.timer
[Unit]
Description=Update Caddy Certificates (Daily)
[Timer]
OnCalendar=*-*-* 01:00:00
Persistent=true
[Install]
WantedBy=timers.target
Just make sure to enable it:
systemctl --user daemon-reload
systemctl --user start --now caddy-lego.timer
Caddy
Now, we need to tell Caddy where to find the certificates. Fortunately, that’s a simple certificate. I also added HSTS and compression.
If you follow my volume above, the location for the certificates will be in /certs/certificates/ IN the Caddy container. In your actual server, it will be in ~/containers/caddy/lego/certificates. Ensure you get the correct one because your browser will spew out errors otherwise.
(certificate) {
tls /certs/certificates/_.pdlozano.com.crt /certs/certificates/_.pdlozano.com.key
}
(compression) {
encode gzip zstd
}
(hardening) {
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
Referrer-Policy "strict-origin-when-cross-origin"
}
}
(local-wildcard) {
@{args[0]} host {args[0]}.pdlozano.com
handle @{args[0]} {
import hardening
import cache-static
reverse_proxy {args[1]}
}
}
pdlozano.com {
import certificate
import compression
import hardening
reverse_proxy host.containers.internal:8000
}
*.pdlozano.com {
import certificate
import compression
import local-wildcard site1 host.containers.internal:8001
import local-wildcard site2 host.containers.internal:8002
import local-wildcard site3 host.containers.internal:5003
import local-wildcard site4 host.containers.internal:8004
}
Now, a simple restart of the pod should work.
Overall, I dedicated 2 hours of my life to do this to avoid having to rebuild Caddy. I didn’t want to keep up with it so the solution was a simple sidecar service to handle the DNS challenge.