01 / WAR STORIES
The Server That Wouldn't Die
A server can survive far more abuse than anyone expects. This story explores the machines that refuse to disappear: services that respawn after being killed, processes that remain attached to unexpected supervisors, failed shutdowns, and systems that mysteriously return after administrators believe they have been stopped. The investigation leads through process trees, service managers, watchdogs, cron jobs and automation—revealing that sometimes the problem isn't how to kill a process, but discovering what keeps bringing it back.
02 / WAR STORIES
When DNS Wasn't DNS
Few infrastructure problems create as much confusion as a failure that looks exactly like DNS. An application cannot connect, a hostname resolves differently from another machine, and every test appears to point toward the resolver. The real cause may instead be /etc/hosts, split-horizon DNS, stale caches, search domains, IPv6, resolver configuration or an application using a different resolution mechanism entirely. This is a practical investigation into why "DNS is broken" is often only the beginning of the diagnosis.
03 / WAR STORIES
The Friday Kernel Upgrade
Kernel upgrades are routine—until the machine doesn't come back. A seemingly harmless update can expose problems involving bootloaders, initramfs images, storage drivers, filesystem modules, third-party kernel modules or hardware compatibility. This story follows the kind of Friday-afternoon failure that turns a routine maintenance window into an incident, demonstrating why kernel changes require rollback planning, console access, tested recovery procedures and a clear understanding of the complete boot chain.
04 / WAR STORIES
rm -rf: A Career Development Program
Recursive deletion is one of the most powerful and dangerous capabilities available from a Unix shell. A misplaced variable, an unexpected empty value, a symbolic link assumption or a command executed from the wrong directory can turn a routine cleanup operation into data loss. This story examines how destructive commands happen, why shell expansion and quoting matter, and how permissions, backups, immutable infrastructure and automation can reduce the consequences of human error.
05 / WAR STORIES
The RAID Array That Lied
RAID can protect against certain disk failures, but it cannot guarantee data integrity or eliminate operational risk. An array may report itself as healthy while another component is silently failing, rebuilds can expose previously unreadable sectors, and hardware RAID controllers can obscure the actual condition of individual disks. This story explores degraded arrays, SMART data, rebuilds, filesystem corruption and the uncomfortable distinction between redundancy and backup.
06 / WAR STORIES
That One Broken Symlink
A symbolic link is only a reference to another path, but a single broken link can have consequences far beyond a missing file. Applications may fail to start, libraries may no longer resolve, configuration directories can point at unexpected locations and deployment systems can silently install files somewhere other than intended. This story follows the investigation of a deceptively small filesystem problem and the lessons it teaches about paths, symlinks, mount points, package management and system configuration.
07 / WAR STORIES
Production Was Full of Logs
Disk exhaustion is one of the oldest and most reliable ways to bring a production system to its knees. Applications continue writing logs, databases consume transaction or journal space, container runtimes accumulate unused layers, and suddenly a filesystem reaches 100 percent utilization. The resulting failures can be surprisingly broad: services cannot write temporary files, databases stop accepting transactions and even administrators may struggle to log in. This story explores how to identify what consumed the space and how log rotation, retention policies, monitoring and capacity planning prevent a repeat.
08 / WAR STORIES
Why We Don't Edit Production Servers
Manually fixing a production server can be remarkably effective—and remarkably dangerous. A change made directly to a live system may solve the immediate problem while leaving the configuration undocumented, unreproducible and vulnerable to being overwritten by the next deployment or automation run. This story examines configuration drift, undocumented fixes, emergency changes and the transition from "fix the server" to "fix the system." Infrastructure as code, immutable infrastructure, configuration management and Git-based workflows provide the mechanisms for making operational changes repeatable and auditable.