Deploying without an SSH key in CI

Say your image is built, scanned and published on every push (this is how I do it). It sits in the registry; the server is still running last week’s.

The usual way to close that gap is to have CI reach into the server: an SSH key in the CI secrets, a step that connects, runs docker compose pull && docker compose up -d, and disconnects. It works, and it means your CI system holds a key to production. Anyone who compromises a workflow, a leaked token, a bad third-party action, gets a shell on your server with it.

There is another direction. Instead of CI pushing into the server, the server pulls when CI tells it to. CI holds no key, no login, no shell. It holds one secret that can do exactly one thing: ask for a redeploy.

Push versus pull

A push deploy runs the deploy from CI: CI has credentials for the server and acts on it directly. A pull deploy puts a small agent on the server. The agent owns the credentials, watches the registry or waits for a signal, and runs pull and up itself. CI only sends the signal.

Pull deploys split into two flavors. The agent can poll: check the registry on a timer, a cron every few minutes, and redeploy when the watched tag moves. Watchtower works this way, and it needs nothing reachable from outside, no endpoint to expose, which is its charm. The catch keeps it out of my production setups. Watchtower watches by mounting the Docker socket, which is root on the host in all but name, and it deploys on its own timer following whatever tag the container tracks, so the version that goes live and the moment it does come loose from the pipeline that gated and tested the image. Against a moving tag like latest, any push lands unsupervised, and a rollback races the next poll. It fits a homelab; for production I want one tested, pinned build deployed at a time I chose.

The other flavor is a trigger: CI tells the agent the moment a new image lands, and it deploys at once, the exact tag CI built and scanned a moment earlier. That is the one I run. It ties each deploy to the gated build that produced it, and the agent already sat behind the proxy, reachable.

The agent I use is Komodo, which manages compose stacks on the server and exposes a webhook: call it, and it pulls the named image and runs the stack back up. The shape matters more than the tool: the credentials live next to the thing they deploy, not in the system that builds it.

Signing the signal

A webhook that redeploys on request is a webhook anyone who finds the URL can fire. So the request is signed.

Komodo speaks the same protocol GitHub’s own webhooks use. The caller computes an HMAC of the request body with a shared secret and sends it in an X-Hub-Signature-256 header. An HMAC is a fingerprint of the body, mixed with a secret key both sides hold. The agent recomputes it with its copy of the secret and compares. A match proves two things at once: the request came from someone holding the secret, and the body was not touched on the way.

In the workflow, after the publish step:

1
BODY='{"ref":"refs/heads/main"}'
2
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" | awk '{print $2}')
3

4
curl -fsS -X POST "$WEBHOOK_URL" \
5
  -H 'Content-Type: application/json' \
6
  -H "X-Hub-Signature-256: sha256=$SIG" \
7
  -d "$BODY"

The body names the branch, refs/heads/main, which the agent matches against the stack it is configured to deploy. printf '%s', not echo, because a trailing newline would change the bytes and break the signature. The secret lives in the CI secrets and on the server, nowhere else.

The 401 that was not about the signature

The first run of that step returned 401 Unauthorized. The obvious suspect is the signature, so that is where the time went: re-checking the HMAC, the encoding, the trailing newline, whether the secret matched on both ends. All of it was correct.

The tell was in the response body. A signature mismatch from the agent would be a short JSON error. This was HTML, a login page, redirecting to an SSO provider. The request was never reaching Komodo. It was hitting Pangolin, the reverse proxy in front of it, which gates every route behind single sign-on and was bouncing an unauthenticated call to the login screen.

The lesson outlived the bug. When a webhook returns 401, the first question is not “is my signature wrong” but “what answered me”. I had assumed Komodo answered. The proxy did.

The fix is to let the call through the proxy without a browser session. Pangolin, with its extended-compatibility option on, forwards an HTTP Basic auth header to the service behind it. Komodo ignores that header, it does its own check with the signature, but Pangolin sees valid credentials and stops bouncing the request. One flag on the request:

1
curl -fsS -u "$BASIC_AUTH" -X POST "$WEBHOOK_URL" \
2
  -H 'Content-Type: application/json' \
3
  -H "X-Hub-Signature-256: sha256=$SIG" \
4
  -d "$BODY"

Two checks now stack: Pangolin’s basic auth gets the request to Komodo, and Komodo’s signature check decides whether to act on it.

Failing loudly

A deploy step that swallows its result is worse than none. The final version captures the status and the body, prints them, and fails the job on anything that is not a success:

1
STATUS=$(curl -sS -u "$BASIC_AUTH" -o /tmp/resp -w '%{http_code}' \
2
  -X POST "$WEBHOOK_URL" \
3
  -H 'Content-Type: application/json' \
4
  -H "X-Hub-Signature-256: sha256=$SIG" \
5
  -d "$BODY")
6

7
echo "status: $STATUS"
8
cat /tmp/resp
9
test "$STATUS" -ge 200 -a "$STATUS" -lt 300

That last line is what turned the silent 401 into a red check with the login-page HTML printed right under it. Without it, the deploy fails and the pipeline still goes green.

Why this is the safer shape

CI holds no SSH key, no server login, no Docker access to production. The two secrets it does hold, the webhook’s signing secret and the proxy’s basic-auth credential, do one thing between them: trigger a redeploy of whatever tag is already configured. A compromised workflow cannot open a shell on the server, because there is no shell to open. It cannot read the database or the server’s other secrets, because it was never given them.

This is not a force field. An attacker who also takes over the registry push could ship a bad image and then trigger its deploy. But the server’s credentials never sat in CI to be stolen, and the trigger on its own is close to useless: it redeploys the current image, nothing more.

What this costs

You run an agent now, one more service to install, update and watch. The webhook is one more link that can break, and it breaks in its own ways, the 401 above being one afternoon of them. A pull deploy is also a step removed from the action: CI reports success when the signal was accepted, not when the new container is healthy, so the real confirmation lives in the agent’s logs or a health check after the fact, not in the green CI run.

Your setup is not this setup

Komodo and Pangolin are my pieces. A few lines of shell behind a tiny authenticated route do the triggered version; Kubernetes has Argo and Flux watching for new images. The constants are the two ideas worth taking: deploy by pulling, so production credentials never enter CI, and, if you trigger rather than poll, sign that trigger so only CI can fire it. The tools underneath are yours to pick.

How the pieces fit

Put end to end, the moves chain up. A push runs the gate, builds the image, smoke tests it under production flags, scans it, and publishes it; a signed webhook tells an agent on the server, which pulls the image and brings the stack back up on a frozen, unprivileged, port-less container. From a git push to a live deploy, nothing along the way runs as root, ships a build tool, leaks a secret to docker inspect, or hands CI a key to the box.

None of it is a finished recipe. Each piece stands on its own, and each has variants I passed over: a different base, a poll instead of a trigger, a registry that is not GHCR. Take the parts that fit your setup and leave the rest. The point was never this exact stack, it was knowing where each risk went.