Idempotency fixes and split server_config into per-system roles

kglitchy commented

2026-06-05 22:18:19 +00:00

Owner

Two related cleanups to the Ansible repo: a batch of idempotency/correctness fixes, then splitting the monolithic server_config role into per-system roles.

Idempotency & correctness

Caddy handler ordering bug: Restart caddy was defined before Reload systemd daemon, so a quadlet change restarted against the stale unit (handlers run in definition order).
Service-start guarantees: livekit, lk-jwt, valkey, hookshot relied only on restart handlers — a converge with unchanged config never brought a down unit back up. Added explicit enabled + started tasks.
coturn → Podman quadlet: was a native distro package whose Ansible config was silently ignored (a malformed ExecStart override drop-in), running as the turnserver user that couldn't read Caddy's 0600 certs — so TURNS/5349 never listened. Now a rootful quadlet (can read the certs); native unit stopped+masked. ⚠️ Behavioral change to call infra — test a call after deploy.
Firewall uses community.general.ufw instead of command + stdout parsing.
Image pre-pull uses containers.podman.podman_image instead of podman pull every converge.
Declared collection deps in requirements.yml (CI/deploy EE only ship ansible.posix).

Role split

server_config → 9 per-system roles: tuwunel, caddy, firewall, livekit (incl. lk-jwt), coturn, element, valkey, hookshot, stickerpicker.
Shared vars + vault moved to inventory/group_vars/glitches_chat/; inventory converted to YAML under inventory/.
Removed the var-naming[no-role-prefix] skip — the split satisfies it natively (lk_jwt_*→livekit_jwt_*, matrix_hookshot→hookshot).
Vault secrets standardized on a vault_* prefix; template refs updated.

Validation

just check passes (0 failures, 0 warnings).
Full check-mode dry-run against the host: ok=54, failed=0, no undefined vars.

The deployed matrix-hookshot unit/paths intentionally keep the upstream name (renaming them is a separate host migration).

Two related cleanups to the Ansible repo: a batch of idempotency/correctness fixes, then splitting the monolithic `server_config` role into per-system roles. ## Idempotency & correctness - **Caddy handler ordering bug**: `Restart caddy` was defined before `Reload systemd daemon`, so a quadlet change restarted against the stale unit (handlers run in definition order). - **Service-start guarantees**: livekit, lk-jwt, valkey, hookshot relied only on restart handlers — a converge with unchanged config never brought a down unit back up. Added explicit `enabled + started` tasks. - **coturn → Podman quadlet**: was a native distro package whose Ansible config was silently *ignored* (a malformed `ExecStart` override drop-in), running as the `turnserver` user that **couldn't read Caddy's `0600` certs — so TURNS/5349 never listened**. Now a rootful quadlet (can read the certs); native unit stopped+masked. ⚠️ Behavioral change to call infra — test a call after deploy. - **Firewall** uses `community.general.ufw` instead of `command` + stdout parsing. - **Image pre-pull** uses `containers.podman.podman_image` instead of `podman pull` every converge. - Declared collection deps in `requirements.yml` (CI/deploy EE only ship `ansible.posix`). ## Role split - `server_config` → 9 per-system roles: `tuwunel`, `caddy`, `firewall`, `livekit` (incl. lk-jwt), `coturn`, `element`, `valkey`, `hookshot`, `stickerpicker`. - Shared vars + vault moved to `inventory/group_vars/glitches_chat/`; inventory converted to YAML under `inventory/`. - Removed the `var-naming[no-role-prefix]` skip — the split satisfies it natively (`lk_jwt_*`→`livekit_jwt_*`, `matrix_hookshot`→`hookshot`). - Vault secrets standardized on a `vault_*` prefix; template refs updated. ## Validation - `just check` passes (0 failures, 0 warnings). - Full check-mode dry-run against the host: `ok=54, failed=0`, no undefined vars. The deployed `matrix-hookshot` unit/paths intentionally keep the upstream name (renaming them is a separate host migration).

kglitchy added 10 commits

2026-06-05 22:18:19 +00:00

Fix caddy quadlet restart firing before systemd daemon-reload 2a00bb29ca

Handlers run in definition order, not notify order. "Restart caddy" was
defined above "Reload systemd daemon", so a caddy.container change
restarted the service against the stale unit before systemd re-read it.
Move "Reload systemd daemon" to the top of the handlers file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Ensure container services are enabled and started on every converge baa8a9997e

livekit, lk-jwt-service, valkey, and matrix-hookshot relied solely on
restart handlers, so a converge with unchanged config would never bring a
down unit back up while still reporting green. Add explicit
"enabled + started" systemd tasks matching the tuwunel/caddy pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Migrate coturn to a Podman quadlet 01562a71d5

Native coturn.service ran as the turnserver user (couldn't read Caddy's
0600 certs, so TURNS/5349 never came up) and a malformed override drop-in
left it reading the unmanaged /etc/turnserver.conf. Run it as a rootful
quadlet instead; stop+mask the native unit and drop the override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Drop SSH common args from inventory; use agent/default key 108cbd3dfa

Use community.general.ufw instead of raw ufw commands 90e8f5992c

Replaces the command-loop + stdout parsing + slurp/grep dance with the
declarative ufw module. Verified idempotent against the host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Use podman_image module for image pre-pull b0c7c88a09

The raw 'podman pull' commands hit the network every converge (masked by
changed_when: false). podman_image pulls only when the image is absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Split UFW default-policy loop into explicit tasks b038040428

Looping a templated policy defeated ansible-lint's arg validation; two
plain tasks are clearer and lint clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Declare collection deps in requirements.yml 0613b11d40

ufw and podman_image need community.general and containers.podman, which
neither CI (ansible-core + ansible.posix) nor the deploy EE
(community-ee-base) ship. Install all three from requirements.yml in both
Tekton tasks. Locally, collections_path points at ./collections (ignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Split server_config into per-system roles 81bafde25a

One role per service (tuwunel, caddy, firewall, livekit, coturn, element,
valkey, matrix_hookshot, stickerpicker); lk-jwt folds into livekit with
lk_jwt_* renamed livekit_jwt_*. Shared values and the vault move to
group_vars; this lets var-naming[no-role-prefix] be unskipped.

Inventory becomes YAML under inventory/, with group_vars colocated there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Rename matrix_hookshot role and vars to hookshot

Pipelines as Code CI / glitches-chat-test Success

Details

37c893d80b

It's obviously for Matrix. Role matrix_hookshot -> hookshot, vars
matrix_hookshot_* -> hookshot_*, tag/templates likewise. Also point
templates at the renamed vault secrets (vault_tuwunel_registration_token,
vault_hookshot_*). The deployed unit/paths stay matrix-hookshot (upstream
name; renaming them is a host migration). Dry-run clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

kglitchy commented

2026-06-05 22:18:21 +00:00

Author

Owner

Pipelines as Code CI/glitches-chat-test-5qhh7 is running.

Starting Pipelinerun glitches-chat-test-5qhh7 in namespace glitches-chat-pipelines

You can monitor the execution using the Tekton Dashboard PipelineRun viewer or through the command line by
using the tkn CLI with the following command:

tkn pr logs -n glitches-chat-pipelines glitches-chat-test-5qhh7 -f

Pipelines as Code CI/glitches-chat-test-5qhh7 is running. Starting Pipelinerun <b>[glitches-chat-test-5qhh7](https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns/glitches-chat-test-5qhh7)</b> in namespace <b>glitches-chat-pipelines</b> You can monitor the execution using the [Tekton Dashboard](https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns/glitches-chat-test-5qhh7) PipelineRun viewer or through the command line by using the [tkn](https://tekton.dev/docs/cli/#installation) CLI with the following command: <code>tkn pr logs -n glitches-chat-pipelines glitches-chat-test-5qhh7 -f</code>

kglitchy commented

2026-06-05 22:19:27 +00:00

Author

Owner

Pipelines as Code CI/glitches-chat-test-5qhh7 has successfully validated your commit.

Namespace: glitches-chat-pipelines
PipelineRun: glitches-chat-test-5qhh7

Task Statuses:

Status	Duration	Name
Succeeded	23 seconds	fetch-repository
Succeeded	43 seconds	Static checks

Pipelines as Code CI/glitches-chat-test-5qhh7 has <b>successfully</b> validated your commit. <ul> <li><b>Namespace</b>: <a href="https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns">glitches-chat-pipelines</a></li> <li><b>PipelineRun:</b> <a href="https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns/glitches-chat-test-5qhh7">glitches-chat-test-5qhh7</a></li> </ul> <hr> <h4>Task Statuses:</h4> <table> <tr><th>Status</th><th>Duration</th><th>Name</th></tr> <tr> <td>Succeeded</td> <td>23 seconds</td><td> [fetch-repository](https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns/glitches-chat-test-5qhh7?pipelineTask=fetch-repository) </td></tr> <tr> <td>Succeeded</td> <td>43 seconds</td><td> [Static checks](https://tekton.corp.mulliken.net/#/namespaces/glitches-chat-pipelines/pipelineruns/glitches-chat-test-5qhh7?pipelineTask=check) </td></tr> </table>

kglitchy merged commit 066e252e1b into main

2026-06-05 22:19:43 +00:00

kglitchy referenced this pull request from a commit

2026-06-05 22:19:44 +00:00

Merge pull request 'Idempotency fixes and split server_config into per-system roles' (#3) from fix/idempotency-and-service-state into main

Rows
Columns

Idempotency fixes and split server_config into per-system roles #3

Idempotency & correctness

Role split

Validation

Task Statuses: