smoketest

One command answers "does this application actually work?" for CLI, GUI and service installs on a machine where nothing can be installed — an air-gapped Ubuntu, Rocky or RHEL box.

You copy a directory, write a YAML file per application, and run:

$ ./smoketest run --config ./configs/
ripgrep (cli) — configs/ripgrep.yaml
  PASS  present
  PASS  version
  FAIL  searches                  exit 2, wanted 0: README.md: No such file or directory (os error 2)

xclock (gui) — configs/xclock.yaml
  PASS  launches
  SKIP  shuts down cleanly        no display ($DISPLAY unset) and xvfb-run is not installed (install the Xvfb …)

nginx-stack (service) — configs/nginx-stack.yaml
  PASS  start (lifecycle)
  PASS  ready (lifecycle)
  PASS  all containers up
  PASS  port open
  PASS  serves landing page
  PASS  stop (lifecycle)

8 passed, 1 failed, 1 skipped

Every test is PASS, FAIL or SKIP with a one-line reason; the service is guaranteed to be gone afterwards; and a misbehaving application can neither hang the run nor leave processes behind.

Install: copy two things

There is no installer, no root, no package manager, no network.

smoketest/
├── smoketest          # statically linked linux/amd64 binary (CGO disabled)
├── configs/           # one YAML file per application
├── examples/          # the stacks, scripts and server the service configs start
└── README.md          # this file

Build it (Go 1.25+, the only dependency is gopkg.in/yaml.v3):

$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath \
    -ldflags '-s -w' -o smoketest ./cmd/smoketest

…then copy the directory to the target and ./smoketest run --config ./configs/. Run it from the directory holding README.md: configs/ripgrep.yaml greps that file as its "does it actually search" test, so the working directory matters.

Usage

smoketest run --config <file|dir> [options]

  --config PATH        one YAML file, or a directory (every *.yaml / *.yml below
                       it is found and run, in path order)
  --app NAME           run only this app; repeat the flag or write NAME,NAME
                       (the order you name them is the order they run in)
  --json               write the whole result tree as JSON on stdout, nothing else
  --quiet              print failures only (the summary line is always printed)
  --keep-up            leave a started service running for inspection
  --strict             a SKIP fails the run, as does a run in which no test passed
  --results-dir DIR    keep evidence (GUI screenshots, a service's start log) here
  -h, --help / -v, --version

Exit status:

code	meaning
0	every test passed or was skipped
1	at least one test failed — or, with --strict, was skipped, or no test passed at all
2	the tool or a config file is broken — nothing was run

smoketest --version and smoketest --help need no config.

Verdicts

• PASS — the application did what the check asked.
• FAIL — it did not, or it had to be killed for misbehaving (timeout, ignored shutdown, crash, unresolved libraries).
• SKIP — this machine lacks an optional prerequisite: no docker, no xvfb-run and no display, no screenshot tool. A missing host tool never blames the application, and SKIP does not change the exit code. The reason line says what to install.

A run in which no test passed — every one skipped, say, on a box with no docker — still exits 0, because nothing failed; but it says nothing was verified under the summary line ("nothing_verified": true in --json), since it cannot tell you whether anything works. For an acceptance pass, where a check the machine could not perform is a check not done, use --strict: then any SKIP, or a run that verified nothing, exits 1, and --quiet shows the SKIPs too.

Config validation happens before anything runs

Every file in the run is parsed and validated up front. An unknown check name, a check that does not belong to that app type, a regexp that does not compile, a key a check does not read, a typo'd field name, a duplicate app name, or a timeout shorter than the deadline it polls for aborts the whole run with exit 2, naming the file, app, test and field:

smoketest: configuration is not usable:
configs/typo.yaml app "ripgrep" test "version" field "check": unknown check "versionn" for app type "cli" (valid: desktop_file, exit_code, help, ldd, present, stdout_match, version)

Fix the config and rerun. When --config names one file nothing is read beyond that file; when it names a directory, every file in it is read and validated before the first test of any app runs — so a broken file late in the list still means no test executes, but the error report covers all of the files it looked at. Either way, no application is started and no evidence is written until the whole set is valid.

Config reference

One file, one application. Keys in bold are required.

Application keys

key	type	meaning
app	string	name shown in output; unique across the run
type	cli | gui | service	which checks are available
binary	string	executable under test, PATH entry or path (required for cli/gui; a service app must not set it)
tests	list	the checks to run, in order (at least one)
description	string	free text, kept in --json
dir	string	working directory for this app's commands and checks (must exist)
env	map	extra environment variables for every command of this app
timeout	duration	hard per-test cap for this app's tests (default 60s)
grace	duration	SIGTERM-to-SIGKILL wait for anything smoketest stops (default 5s)
setup, teardown	string or list	shell hook commands run before/after the app's tests
run	string	the service itself, kept running and supervised (service apps only)
start, stop	string	one-shot bring-up and shutdown commands (service apps only)
compose	mapping	file (path or list), project, manage — a Docker Compose stack smoketest brings up and down (service apps only)
ready	mapping	readiness gate polled after the bring-up (service apps only)
desktop_file	string	default .desktop path for the desktop_file check
screenshot	bool	capture GUI evidence for this app's GUI checks (gui apps; default off) — needs --results-dir

Test keys

Each test needs name and check; everything else is read only by the checks that understand it (the table below lists each check's keys). A key set for a check that ignores it is rejected up front.

key	type	meaning
name	string	shown in the table; unique within the app
check	string	one of the check types below
timeout	duration	hard wall on this test (default: the app's, else 60s)
description	string	free text, kept in --json

Check types

cli and gui apps may use the command checks; gui adds the GUI ones; a service app uses the service checks, plus exit_code and stdout_match — which, since a service has no binary, name their program with path (for example path: systemctl, args: [is-active, myapp], or path: nginx, args: ["-t"] to check its configuration).

check	keys it reads	passes when
present	path	the binary resolves in PATH (or at path) and is executable
exit_code	args, expect_exit, path, dir, env	running binary args… exits expect_exit (default 0)
stdout_match	args, pattern, expect_exit, path, dir, env	exit is expect_exit (default 0) and stdout matches the regexp pattern
version	flag, args, path, dir, env	binary --version (or flag) exits 0 and prints something
help	flag, args, path, dir, env	same with --help
ldd	path	ldd output has no not found (a statically linked binary passes; no ldd on the host → SKIP)
desktop_file	file, path	a .desktop entry exists and parses: a [Desktop Entry] section with Name= and a non-empty Exec=
launch_alive	args, seconds, path, dir, env, screenshot	the app starts and is still running after seconds (default 3s)
clean_shutdown	args, seconds, path, dir, env, screenshot	SIGTERM stops the app within the test's remaining budget (less grace), and anything it started within grace after it, with no core dump and no crash signal
screenshot	dir, env	a screen grab is written to --results-dir using whichever of import/scrot/grim exists (none → SKIP)
port_open	host, port, deadline	a TCP connect to host:port succeeds within deadline (default 10s)
http_ok	url, expect_status, body_match, deadline	GET url returns expect_status (default 200) and, if given, the body matches the regexp body_match — retried until deadline
systemd_active	unit, deadline, seconds	the systemd unit is active within deadline (default 10s) and, with seconds, stays active with no restart for that long — a unit in a restart loop is active for a moment at a time; failed, masked or not installed FAIL at once (no systemd running, as in a container → SKIP)
compose_running	deadline (stack from the app's compose:)	every container of the stack is running — none restarting, exited, paused or merely created (no docker → SKIP)
container_healthy	deadline (stack from the app's compose:)	every container that declares a HEALTHCHECK reports healthy; if none declares one the check passes and says so; a container that is not up is reported as that, not as "unhealthy" (no docker → SKIP)

Durations accept 45s, 2m30s, 1500ms, or a bare number of seconds (15). Empty strings are rejected in args, setup, teardown and env keys: an argument that vanishes on the way to the application is never what was meant.

Service lifecycle

A service is brought up in exactly one of three ways, and the config says which:

# 1. run: the service itself, a foreground process smoketest supervises
run: python3 /opt/app/server.py --port 9000

# 2. start: a one-shot command that brings the service up and returns
start: /opt/app/bin/appctl start
stop:  /opt/app/bin/appctl stop        # optional, with run or start

# 3. compose: a stack smoketest brings up and down itself
compose:
  file: /opt/app/docker-compose.yml    # one path or a list (-f, in order)
  project: shop                        # optional (-p)

followed, in any of the three, by an optional readiness gate and the tests:

ready:
  check: port_open          # any service check may gate readiness
  host: 127.0.0.1
  port: 8080
  deadline: 90s
tests:
  - { name: port open, check: port_open, host: 127.0.0.1, port: 8080 }

• run is launched detached and must keep running for the whole section. If it exits — before readiness, during a test, or after the last test — the app FAILs with a service (lifecycle) row giving its exit status and the last line it printed, and the tests still to come are SKIPped for that reason rather than polling a dead service until their deadlines.

• start must exit, and its exit status is its verdict: 0 is a PASS, anything else a FAIL quoting its output. One that is still running after the app's timeout (else 120s) is killed and FAILs with a reminder that a command which is the service belongs in run:. Whatever start launched — a daemon that double-forked and went setsid included — is tracked, and torn down after stop.

• compose runs docker compose -f … -p … up -d before the tests and … down after them, in the app's dir. compose_running and container_healthy ask about exactly that stack — they need a compose: block, and nothing is inferred from command lines. Add manage: false to verify a stack somebody else runs: smoketest then never brings it up or down.

• A stack your own scripts bring up. Some installs ship a start.sh that generates its compose file and runs docker compose up itself, and only works from its own directory. Give the scripts as start/stop, the directory as dir, and name the stack with an unmanaged compose: block — the scripts bring it up and down, and the block tells the compose checks what to ask about:

  dir: /opt/vendorapp          # start, stop and the compose checks all run here
  start: ./start.sh            # generates docker-compose.yml, runs `up -d`
  stop: ./stop.sh
  compose: { file: docker-compose.yml, manage: false }
  

  The compose file only has to exist by the time the checks run. On a machine with no docker this SKIPs like any compose stack, and neither script is run. On a machine with no docker (or no compose subcommand) the bring-up and every compose check SKIP; the host is what is missing, not the application.

• run, start and stop are shell command lines, so cd /srv/app && ./ctl start works. All three run inside the app's sandboxed environment and dir.

• ready is polled after the bring-up and is reported as its own ready (lifecycle) row. If readiness fails the app's tests still run and each reports its own reason — you see the whole picture, not one error.

• Teardown is guaranteed. After the tests — whatever failed, and even after Ctrl-C — smoketest runs stop (or compose down), then kills anything the bring-up started that is still there (cleanup (lifecycle) says so), then runs the teardown hooks. A run service with no stop is stopped with SIGTERM, then SIGKILL after the app's grace (default 5s).

• The ports must be free first. port_open and http_ok cannot tell the service smoketest launched from anything else listening on the same address. So before run, start or compose up, every address on this machine that the ready gate and the tests probe is tried; if one already answers, start FAILs naming the address and, when /proc can tell, the process holding it, and the tests are SKIPped — rather than a service that never came up PASSing on someone else's listener. Verify-only configs are exempt: there the port is supposed to be in use.

• Launching nothing means verify-only. A service config without run, start or a managed compose probes a service somebody else brought up and never stops it; stop or ready without something to stop or gate is rejected as a config error.

• --keep-up suppresses all of the above and prints the command to stop the service by hand, with the kept service's pid and process group. The throwaway sandbox is still removed — --keep-up keeps the application, not the temporary home — so pass --results-dir as well if you want the service's output kept: it is copied to <results-dir>/<app>/start.log. launch_alive and clean_shutdown always stop the GUI process they started, whatever flags were given, because a smoke test must not leak an application into the operator's session.

• The stack is listed with docker compose ps --all, so a container that has exited counts against it. That is deliberate: a stack whose web container died an hour ago still answers docker compose ps with an empty "running" list, and "does this application work?" deserves the answer no. A compose file named by the config that does not exist on disk is a FAIL naming that path at once — an incomplete installation is the answer the tool was asked for, not a reason to SKIP.

desktop_file and the real user's directories

desktop_file looks in the operator's $HOME/.local/share/applications, $XDG_DATA_DIRS/applications (default /usr/local/share:/usr/share) and /usr/{,local/}share/applications — deliberately not in the throwaway sandbox — because the question is whether the installation registered a launcher. Give file: when the entry is not called <binary>.desktop.

Environment, $VAR and the sandbox

Every app's tests run with a private HOME, XDG_CONFIG_HOME, XDG_CACHE_HOME, XDG_DATA_HOME, XDG_STATE_HOME and XDG_RUNTIME_DIR under a /tmp/smoketest-<app>-… directory, removed when the app's section ends — on every path, including --keep-up and a cancelled run; nothing of smoketest's survives the invocation except the application you asked it to leave running and the evidence in --results-dir. The layers, later winning: the environment smoketest itself has, then the sandbox, then the app's env:, then a test's env:. Values may reference variables with $VAR or ${VAR}, expanded against the layer they join — so PATH: "$HOME/bin" :$PATH inside an app's env: already means the sandbox home.

Because the app cannot write into the real user's configuration, a bad install cannot poison the box: this is also where clean_shutdown looks for core dumps.

Timeouts

Three keys decide every duration; everything else is derived from them.

key	where	meaning
timeout	test or app	the hard wall on one test (default 60s); also caps start, stop, setup and teardown (default 120s)
deadline / seconds	test	the window the check itself observes: how long it polls, how long it watches
grace	app	how long anything smoketest stops gets between SIGTERM and SIGKILL (default 5s)

• The wall holds. A check spends at most timeout − grace waiting on the application, so whatever it has to stop at the end — a command at its timeout, an app that ignored SIGTERM — has been SIGKILLed inside the wall. On expiry the test FAILs with timed out after 60s; the whole process group was killed; the command and everything it started got SIGTERM, then SIGKILL grace later. "Everything it started" is wider than its process group — see below.
• The window fits. A test with no timeout of its own inherits the app's, else 60s, and when nothing explicit was set it is widened to window + grace + 10s, so the check gets to report its own deadline rather than being killed mid-poll. A timeout that is set and is too short for its window — under the deadline, under deadline + seconds for systemd_active, or under the seconds a GUI check watches including the 3s default — is rejected as a config error rather than silently shortened, because the only thing such a test can report is a kill the tool caused.
• grace is a promise to the application. A run service stopped at teardown, a stop command's service winding down, a child left behind by a GUI app on SIGTERM: each gets grace before SIGKILL. A service that needs longer to shut down cleanly — a database flushing to disk — needs grace raised, or teardown kills it mid-flush.
• If a check somehow refuses to unwind, a watchdog reports a timeout for it at timeout + grace + 2s and moves on — a hung application can slow the run by a bounded amount and no further.

Nothing is left behind

A process group alone does not contain an application: a daemon that calls setsid() leaves it, and a group kill misses it. So every command smoketest runs carries a SMOKETEST_FAMILY tag in its environment, which its descendants inherit through fork, exec and setsid alike, and smoketest makes itself the subreaper for its descendants, so an orphan is re-parented to smoketest rather than to init. Killing a command kills its process group and every process carrying its tag — wherever it has moved — and a check whose program exits leaves no helper running after it. At the end of each app's section anything still descended from smoketest (something that left its group and cleared its environment) is killed too, and reported in a strays (lifecycle) row. A service kept with --keep-up is not swept.

JSON output

--json writes the tree and nothing else, so it pipes straight into jq:

{
  "tool": "smoketest",
  "version": "1.0.0",
  "started_at": "2026-10-05T14:22:03Z",
  "seconds": 2.41,
  "apps": [
    {
      "app": "nginx-stack",
      "type": "service",
      "config": "configs/nginx-stack.yaml",
      "tests": [
        { "name": "start", "check": "start", "status": "PASS",
          "reason": "/usr/bin/docker compose -f examples/nginx-stack/docker-compose.yml up -d exited 0 after 1.204s: …",
          "seconds": 1.204, "lifecycle": true },
        { "name": "serves landing page", "check": "http_ok", "status": "PASS",
          "seconds": 0.011 }
      ],
      "summary": { "passed": 2, "failed": 0, "skipped": 0, "total": 2 }
    }
  ],
  "summary": { "passed": 2, "failed": 0, "skipped": 0, "total": 2 },
  "interrupted": false
}

reason is present for every result including passes (the table hides it for PASS), evidence lists files a check wrote under --results-dir, and lifecycle marks the setup/start/ready/service/stop/cleanup/strays/teardown rows. Screenshots are opt-in (screenshot: true) and are attached to the verdict they were taken under — a failed clean_shutdown carries the picture of the application that failed to close. Two apps in one run must not share a name: a duplicate is a config error, because --app NAME and --json consumers need one name to mean one application.

Behaviour that makes it safe to point at an unknown install

• Nothing runs without a timeout. Every external command — hooks, lifecycle commands, ldd, the app itself — runs under a deadline, and killing means killing everything it started, daemons that left its process group included.
• A cancelled bring-up is not a verdict on the application. If the run is interrupted while a service is starting, the start row is a SKIP, its ready gate and tests are skipped with the same reason, and teardown still runs against whatever the start command had begun.
• Ctrl-C cancels, teardown still runs. The first interrupt stops launching new tests (they report SKIP: the run was cancelled), the current check is cancelled — its own row is a SKIP, not a FAIL, because the operator ended the test rather than the application failing it — and stop plus the teardown hooks still execute before the process exits. The report says interrupted.
• A missing host tool is never the app's fault: docker, xvfb-run, a display and screenshot tools are all optional and SKIP with a stated reason.
• launch_alive on a headless machine runs the app under xvfb-run -a when it is installed, so X11 apps get a real answer; otherwise SKIP.
• clean_shutdown accepts an app that has no SIGTERM handler. Most X11 applications, xclock among them, let the default disposition stop them: dying from the SIGTERM it was sent, with no core dump, is a PASS. A crash signal, a non-zero exit, a core dump in the sandbox, or an app that ignores SIGTERM entirely is a FAIL.
• The tool makes no network calls of its own. port_open and http_ok connect only to the host/port/url a config names — by convention loopback on the machine under test. Nothing else in smoketest resolves a name, fetches a page, or phones home; Docker is used strictly through the pre-installed docker CLI.

Examples

configs/ ships five working files: ripgrep.yaml (cli), xclock.yaml (gui, headless-safe), nginx-stack.yaml (a Docker Compose stack), vendor-stack.yaml (a compose stack the application's own start.sh/stop.sh generate and bring up, from their own directory) and report-server.yaml (a bare long-running process). Run them from this directory and all five are decided on their merits: xclock PASSes with a display or xvfb-run, SKIPs without either. The service examples start what ships in examples/: nginx-stack brings up examples/nginx-stack/docker-compose.yml, vendor-stack runs examples/vendor-stack/start.sh (which writes its own compose file and refuses to run from anywhere but its directory) — both need the nginx:stable image already loaded, docker load -i nginx.tar on an air-gapped box — and report-server runs examples/report-server/server.py, which needs only python3. Edit compose.file, dir or run to name your own stack, scripts or server — an installed stack usually lives somewhere like /opt/app/ — or copy the file and change app: to keep both.

A compose stack on a machine with no docker CLI SKIPs instead of FAILing: the host is what is missing, not the application.

To write a config for something new, start from the closest example and keep only the tests you want. smoketest run --config myapp.yaml is the whole loop.

Tests

$ go test ./...

The suite covers config parsing and every validation rule, each check type against good and bad fixtures, kill semantics (including a grandchild that survives its parent, a SIGTERM-ignoring daemon that went setsid, one that also cleared its environment, and an app that ignores SIGTERM), teardown on failure, --keep-up, cancellation, sandbox isolation, and the CLI's exit codes. Docker is exercised through a scripted fake, so the compose checks are tested without a daemon; GUI checks use shell fixtures and a stub xvfb-run for the same reason.

Every target distro

$ scripts/matrix.sh                 # or: scripts/matrix.sh rocky9 ubuntu2404

builds the binary and every package's test binary once, then in an Ubuntu 24.04, Ubuntu 22.04, Rocky 9, Rocky 8 and RHEL 9 (UBI) container — as an unprivileged user, with no Go toolchain there — runs the shipped configs from a copy of the delivery directory, runs xclock again with xvfb-run hidden (it must SKIP, not FAIL), and runs the test binaries. It ends with a table of every verdict per distro, marking any that differ. Reports land in dist/matrix/. Building the images downloads packages, so this is for a development machine, not the target. The compose example SKIPs inside the containers, which have no docker; it is exercised by running configs/ on the host. One difference is expected: UBI's repositories carry no X11 packages, so on ubi9 the xclock example FAILs with "xclock" is not on PATH — correctly, since it is not installed there.