Fleet management

Ansible is the primary way to manage collectors, patching and container updates across the fleet. scripts/lxc-install.sh collector is a legacy bootstrap path.

New to Ansible?

This page assumes you know what an inventory, a playbook and a role are. If not, read Getting started first; it takes about fifteen minutes and uses examples from this repo.

cd ansible
ansible-playbook playbooks/site.yml                  # everything, in order
ansible-playbook playbooks/site.yml --limit rpi5     # one host
ansible-playbook playbooks/site.yml --limit pi       # one group
ansible-playbook playbooks/site.yml --check --diff   # dry run

Safe to re-run. A clean fleet reports changed=0 everywhere.

What this manages

monitored is bifurcated by platform, then by distro family. Every monitored host sits in exactly one leaf.

Group	Hosts	Collector	Auto-patched	Container updates
lxc → lxc_debian	tailscale-router, plex, memory, monitor-lxc	yes	yes, central see below	memory only
vm → vm_debian	ubuntu-dev, ubuntu-ai, cloud-services, matrix	yes	yes	yes
pi → pi_debian	rpi5, rpi4b	yes	yes	rpi5 only

Today every leaf is _debian. That is the point of the split rather than an argument against it: a future vm_fedora slots in beside its sibling without disturbing anything, and --limit vm keeps meaning "every VM". Variables divide along the same seam, and deeper groups win:

• platform (lxc / vm / pi): true because of the hardware or the virtualisation. SD-card IO on the Pis. Shared kernel on the containers.
• distro (*_debian): true because of the package manager. Anything naming an apt package belongs here, because that is exactly what a _fedora sibling would not share.

_debian means the Debian family

Four members of vm_debian actually run Ubuntu 26.04. They are grouped with Debian because the axis every role in this repo cares about is apt-vs-dnf, and Ubuntu is on the apt side of it. Reserve a new leaf for distros that would genuinely break these roles.

Read the platform off the host, never off its name

Two hosts are not what they look like. tailscale-router is an LXC, not a VM: it reports a -pve kernel only because a container shares the hypervisor's kernel, and it has no /etc/pve. matrix is a VM, not a container, despite sitting on the LAN beside the LXC guests, which is why it is the one guest with a real sudo setup. Check before you move one: ansible <host> -m setup -a 'filter=ansible_virtualization*'.

Overlay groups

These cut across the platform tree because they answer different questions. A host is in as many as apply.

Group	Hosts	Answers
lan_guests	plex, memory, matrix	how it is reached: ProxyJump via lan_jump_host
central	monitor-lxc	what it runs: the monitoring stack itself
autoupdate	all ten	whether it is auto-patched

Do not fold an overlay into the platform tree. lxc_debian holds both tailscale-router (reached directly over the tailnet) and monitor-lxc (reached through the jump host), so "is a container" and "needs a jump host" are genuinely independent facts. lan_guests are Proxmox guests that are not Tailscale members; they still push telemetry to the same public ingest endpoint as everything else.

The one rule that decides patching

Group membership, never a role conditional

Whether a host is auto-patched is decided solely by its membership of the autoupdate group in the inventory. The unattended_upgrades role has no internal host conditional, because a conditional inside a role is one typo away from auto-upgrading a hypervisor.

The docker_updates role does check for a Docker socket, but that is a capability gate, not a policy gate: it decides whether the role is even applicable, not whether the host is allowed to update.

The central node is a special case

On monitor-lxc, OS and security patches apply automatically, but the monitoring stack itself is upgraded deliberately with scripts/lxc-update.sh central, which re-renders the configs in the same run.

Otherwise a 03:00 unattended run could:

• restart Grafana and Loki, blinding the fleet at exactly the wrong moment
• reload nginx, briefly 502-ing every collector's ingest path
• upgrade a package whose config file this repo owns, without re-rendering it

playbooks/update-metrics.yml runs there too, so anything held back still shows up in apt_upgrades_pending. The blacklist suppresses installs, not visibility.

Deliberately out of scope

Documented so the omission is a decision, not a gap.

qbittorrent (LXC 103)

Dropped for two independent reasons. Its sshd rejects every key we hold, so it was never onboarded and has no collector. And its data filesystem is expected to run at ~100% full, which made the disk alert permanently noisy. While it was still in monitored it also sat in the host-down or-chain, so it fired Host Down every minute forever.

To bring it back: install a key, re-add it under lxc_debian, lan_guests and autoupdate, and give it a mountpoint exclusion in rules-resources.yaml before you do, or you will re-create the disk noise.

t7920, the Proxmox host itself

All of its guests are monitored, but the hypervisor is not. Accept the consequence: host-level CPU, RAM, disk and ZFS pressure on the machine everything else runs on is invisible here. If a guest looks starved, there is no fleet metric explaining why. Watch it in the Proxmox UI.

OPNsense (VM 102, FreeBSD)

Monitor via the os-node_exporter plugin if you want it, scraped rather than pushed. Firmware updates stay manual: an unattended firewall upgrade takes out remote access to everything else.

Others

• arr-stack (LXC 104): stopped in Proxmox.
• win11 (VM 108): would need windows_exporter, a separate path.
• skullsiants, expl01t: pentest tooling is deliberately version-pinned.
• Macs, iPhone: excluded by request.

Inventory layout

ansible/inventory/
  hosts.local.yml        real site data, GITIGNORED
  hosts.example.yml      committed placeholders
  group_vars/
    all/main.yml         shared paths and endpoints
    all/vault.yml        ansible-vault, committed encrypted
    pi/main.yml          SD-card IO tuning (platform)
    pi_debian/main.yml   RPi kernel/bootloader blacklist (distro)
    central/main.yml     loopback ingest, stack blacklist
    lan_guests/main.yml  ProxyJump defaults
  host_vars/
    monitor-lxc.yml         pins MONITOR_HOSTNAME
    tailscale-router.yml    PVE-safe overrides, small-rootfs caps

lxc, lxc_debian, vm and vm_debian have no var files yet. They exist to be extended: the platform/distro seam is where the next override goes, rather than into a host_vars file that quietly grows a second copy of the same setting.

Why tailscale-router is host_vars, not a group

Those overrides used to be group_vars/proxmox, a group of exactly one host described as "hosts running a PVE kernel". That was true but misleading, and once monitored split into lxc/vm/pi a third thing called proxmox sitting beside them was actively confusing. Every value in it is justified by that host's 2.0 GB rootfs and exit-node role, so it belongs to the host. Promoting any of it to lxc would be a real change for plex, memory and monitor-lxc.

The repo is public

All real domains, IPs and usernames live only in hosts.local.yml, which .gitignore excludes via ansible/**/*.local.yml. Note the pre-existing *.local pattern matches only names ending in .local, which would not catch secrets.local.yml.

Variable precedence gotcha

group_vars/<group>/main.yml beats vars set in the inventory file itself. A placeholder left in group_vars/lan_guests silently overrode the real lan_jump_host from hosts.local.yml and cost an afternoon. If a value is site-specific, put it in hosts.local.yml and keep it out of group_vars.