Ansible CI estate
Five roles, seven playbooks and an executable inventory, with validation that fails the build.
Evidence tier: Verified | code and local run. The code exists in the estate repository, and the validation gate has been run locally at pinned tool versions.
This page is a placeholder. The full write-up will cover the role decomposition, the CI gate that fails the build, and the three operational failures the gate was built to catch. What follows is what can be stated today without overclaiming.
What exists
- Five roles (
common_debian,docker_host,lab_management_bootstrap,operational_host,ssh_security), seven playbooks, and an executable inventory. - A validation gate that hard-fails: YAML and
ansible-lintescalation, inventory parsing, two-tier secret scanning, and a regression guard for the lint configuration itself. - Push-to-deploy pipelines across five hosts on a self-hosted runner, with the operational gotchas documented in the repository (a runner killing its own job, a secret whose newline was mangled, a reload endpoint that accepted a request and did nothing).
What Broke / Challenges
A deploy that killed its own runner. The push-to-deploy pipeline runs on a self-hosted runner, and that runner lives in a container on one of the hosts the playbook configures. A play that restarted the container runtime took the runner down mid-job, so the deploy failed at the point where it had already changed the host. The fix was to scope which hosts a given play may touch from that runner and to treat “restart the thing that is executing me” as a first-class hazard in the role, not as an accident.
A secret that arrived one character wrong. A credential written through the pipeline came out with its trailing newline mangled, so authentication failed with a message that pointed at the credential being invalid rather than at its shape. The fix was to normalise and verify secret content at write time, and to record the failure mode in the repository, because the next person to see “invalid token” would otherwise rotate a perfectly good key.
A reload endpoint that accepted the request and did nothing. The service returned success and kept serving the old configuration. From the pipeline’s point of view the deploy had worked. This is the same class of failure as a firewall rule that is present and inert: the control exists, reports success, and has no effect. The play now asserts the resulting state rather than the HTTP status.
A “temporary” lint relaxation that became permanent. A rule was disabled to unblock one change; it stayed disabled for months, and the gate that was supposed to catch regressions could no longer see the class of problem it was built for. The response was a regression guard for the lint configuration itself: a known-bad setting reappearing now fails the build.
What is not claimed
- This is a lab-scale estate: five hosts, one operator. Nothing here is a claim about fleet scale.
- Restore is not rehearsed with a measured RTO. Backup jobs are evidenced; a timed restore is the control that would prove them, and it is not there yet.
- The CI gate is also the reviewer, because there is one operator. A gate that is subtly wrong has no second pair of eyes; that is a known limitation, not a solved problem.
- No public repository. The estate repository is private, so there is no source link on this page. Publishing it would disclose hostnames and network topology; the write-up is the artefact instead.