InSAR Infrastructure Ops

The InSAR processing fleet (3 compute nodes + 4 storage systems) was managed by hand over SSH, so mounts and config drifted asymmetrically across nodes and disk exhaustion only surfaced after the fact. It needed to be standardized, automated, and observable — and to become a stepping stone toward a K8s transition.

Following a 'move real operations, not a toy project' principle, I instrumented and automated the live fleet in place. (1) Ansible control (.173 hub) — manages heterogeneous, unprivileged, and offline nodes via roles/playbooks, pushing code over LAN to the node with no internet. (2) Monitoring stack — Prometheus·Grafana·Alertmanager + 5 exporters (node·windows·snmp) stood up with zero downtime, alerting on disk exhaustion before the auto-eviction fires. (3) Semaphore (Ansible web UI) runs playbooks as buttons with history. (4) NAS mount standardization — audited every code reference before changing paths to prove the blast radius, then unified to /mnt/nas249 NFS additively (zero downtime).

Caught disk exhaustion before the evictor fired (a 🔴 98.9% alert on .64 verified across two inboxes), unified per-node mount asymmetry (/mnt/nas · /mnt/web · /mnt/test249) onto /mnt/nas249 NFS with zero downtime, and ran/verified ping and node_exporter deploys as Semaphore buttons. Aware of the single-hub (.173) SPOF, I'm preparing the next step — distributing it via K8s/virtualization.