The smoke test that tested the wrong container

The image was green on my machine. Seven pages returning 200, the health check passing, the container coming up clean every time I ran it. I pushed, and CI turned red on that same image.

The app behind this is a server-rendered site. A Node process renders the pages, nginx sits in front of it as a reverse proxy, and I run both in one container to ship a single image instead of two. So the container holds two processes that can fail on their own.

The symptom

The smoke test in my pipeline starts the container, waits for its built-in health check to report healthy, then curls a couple of pages. This run, the container never went healthy. One line in the logs said why:

1
nginx: [emerg] open() "/tmp/nginx.pid" failed (13: Permission denied)

Node had started fine, Listening on http://127.0.0.1:3000 printed right above it. nginx was the one refusing to boot. Since nginx answers the health probe, one dead process marked the whole container sick.

nginx writes a small file when it starts, its own process id, to /tmp/nginx.pid. /tmp is the scratch folder every Unix system keeps around, the one place a program running as an ordinary user can always write. Here nginx ran as an ordinary user and could not write to it.

It works on my machine

The same image and the same command worked on my laptop. The difference was a pair of flags I had stopped noticing.

In production the container runs locked down: its root filesystem is read-only, frozen so nothing can write to it, with a small in-memory scratch disk mounted over /tmp for the few files it does need to write. My local runs did the same through the compose file. So on my laptop /tmp was that fresh in-memory disk, writable by anyone, and nginx was happy. The image’s own /tmp never came into the picture.

The CI smoke test mounted no such disk. It ran the container bare, docker run with nothing extra, so nginx fell back to the /tmp baked into the image. And that /tmp, in the image CI had built, carried the wrong permissions. An ordinary user could no longer write to it.

Why the image differed

Same Dockerfile, same base image, two different results. The reason sits in the build engine.

Docker can build images with more than one backend, and the one running in CI was not the one on my laptop. When the CI engine packed the finished image, it stored /tmp without the open-to-everyone permission the base image ships with. My local engine kept that permission. One folder, one bit, present in one build and missing from the other. That is the whole gap between green and red.

The fix I almost shipped

I could have stopped at a one-line patch, force the permission back in the Dockerfile, move on. I did add that line. On its own it would have buried the real problem.

My smoke test was starting the container in a shape it never ships in. Production freezes the filesystem and mounts the scratch disk. The test did neither. So it failed on a problem production does not have, because production mounts a fresh disk over that /tmp and the broken permission underneath stops mattering. And while the test missed nothing real here, it was also never exercising the frozen filesystem, the one constraint most likely to break a container that looks fine on a developer’s writable machine.

The fix was to run the container in the test the way it runs in production:

1
docker run -d --name app \
2
  -p 8080:8080 \
3
  -e ORIGIN=http://localhost:8080 \
4
  -e API_KEY=ci-dummy \
5
  --read-only --tmpfs /tmp:size=64m,mode=1777 \
6
  --cap-drop ALL --security-opt no-new-privileges \
7
  app:ci

--read-only freezes the root filesystem. --tmpfs /tmp mounts the in-memory scratch disk, and mode=1777 gives it the permission that lets any user write. --cap-drop ALL strips every Linux privilege the container does not need. These are the flags production uses, so the test now passes or fails for the reasons production would.

I kept the one-line patch as well, chmod 1777 /tmp in the Dockerfile, so the image is correct even when someone runs it bare. Two layers, no trust placed in either alone.

What I took from it

The bug was not about a permission bit. It was about a test that ran the container in conditions no user ever meets. A green check tells you something only when the thing it checked matches the thing you ship. Run the artifact the way it will run, frozen filesystem, dropped privileges, scratch disk and all, or the check is grading a container that lives only in CI.