Topic 04
Storage tells the truth, but usually one level below where you are looking. The summary line is almost always healthier than the devices under it.
Failure patterns
Each one happened. Each one showed green while it was happening.
What you saw
The pool reports ONLINE. The dashboard tile is green.
What was happening
The pool was online because redundancy was covering a device that had been throwing errors for weeks. Degraded-but-serving looks exactly like fine until the second disk goes.
Read the per-device detail, every time
zpool status -v # error counters live per device, not in the summary line
A pool stays ONLINE right up until it does not.
What you saw
The alert named a disk. The disk it named was healthy when checked.
What was happening
The errors were recorded against a short device name, and short names are assigned at boot. After a reboot the same letter pointed at a different physical disk. The healthy one got investigated and the failing one did not.
Import and alert by stable identifier
zpool import -d /dev/disk/by-id ls -l /dev/disk/by-id/ # by-id survives a reboot, sdX does not
A disk name that moves will send you to the wrong disk.
What you saw
Node exporter shows plenty of free space across the host.
What was happening
The default exporter config excludes some mount paths. The filesystem that filled up was one of the excluded ones, so it simply never appeared in a graph or a rule.
List what is being scraped, not what exists
# what is the exporter actually reporting? curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes # compare that list against df -h
A filesystem that is not scraped cannot run out of space on a graph.
What you saw
Used space jumped, or did not drop after a cleanup.
What was happening
Hardlinked trees count once or many times depending on the flag, and some filesystems report provisioned size rather than consumed size. The number was real; it was answering a different question.
Know which question the tool is answering
du -sh --apparent-size . # provisioned du -sh . # on-disk blocks # across hardlinks these differ by a lot
Check what the measurement measured before you act on it.
House rules: no real addresses, no real hostnames, no topology — every screen is generic, scale figures only. No sponsors. Open-source tools get named; anything for-profit stays generic. Every failure here happened in this rack.
That is the whole channel.