01
The healthcheck that dials itself
What you saw
The container is healthy. Docker says so, your dashboard says so, and the uptime graph is a flat green line.
What was happening
The healthcheck was pointed at the container's own loopback. The VPN tunnel behind it had been down for days. The check passed the whole time because it never left the box.
Point the probe at something outside the box
# gluetun: dial real addresses through the tunnel
HEALTH_TARGET_ADDRESSES=cloudflare.com:443,github.com:443
A healthcheck that dials itself proves only that it is alive.
02
active (running) is not working
What you saw
systemctl says the unit is active (running). It has been active for 41 days.
What was happening
ActiveState only describes the main process. The worker thread inside it had died. The process stayed up, did nothing, and systemd had no opinion about that.
Read SubState, and make the unit prove it is alive
systemctl show -p ActiveState,SubState ingest.service
# in the [Service] section, with Type=notify
WatchdogSec=120
Alert on what came out, not on whether the unit is running.
03
The metric that stopped arriving
What you saw
No alert fired. The threshold rule on that metric has never once gone red.
What was happening
The exporter stopped publishing the series. A metric that is absent cannot cross a threshold, so a rule written as a comparison can never fire again. Silence read as health.
Alert on absence and on staleness, not only on value
# absent() catches a series that vanished
absent(scrape_duration_seconds{job="zfs"})
# staleness catches one that is arriving but stale
time() - job_last_success_timestamp_seconds > 3600
A metric that stopped reporting can never cross a threshold.
04
Probing the address instead of the port
What you saw
The blackbox probe is green. The host answers.
What was happening
The probe was checking that the address responded, not that the service was listening. The host was fine. The service on it had not accepted a connection in hours.
Probe the port the service actually listens on
nc -z -w2 10.0.0.41 443
# blackbox_exporter
modules: { tcp_connect: { prober: tcp } }
Reaching the machine is not reaching the service.
House rules: no real addresses, no real hostnames, no topology — every screen is
generic, scale figures only. No sponsors. Open-source tools get named; anything for-profit stays
generic. Every failure here happened in this rack.