BREAK IT.
FIND IT. FIX IT.
Practical infrastructure failure scenarios. No multiple choice. No magic answers. Follow the evidence.
Troubleshoot the problem. Don't guess the answer.
OBSERVE
Start with the symptoms. What is actually broken?
INVESTIGATE
Use commands, logs, metrics and system state to gather evidence.
DIAGNOSE
Identify the first broken assumption rather than stopping at the visible error.
FIX
Apply the smallest appropriate correction and verify the system.
If you cannot explain why the system failed, you haven't finished troubleshooting.
Choose a failure.
Broken DNS
The application is reachable by IP address but the hostname does not resolve.
Disk at 100%
The server is running out of space, but deleting obvious files doesn't solve it.
CrashLoopBackOff
A Kubernetes application starts, then immediately dies.
Broken systemd Service
The service is enabled, but refuses to stay running.
TCP Connection Failure
The application says the database is down. The database says it isn't.
Terraform Drift
The cloud infrastructure no longer matches what the repository declares.
Kubernetes Node NotReady
Workloads are failing because one cluster node has disappeared from service.
Certificate Expired
The application suddenly reports TLS errors after months of working normally.
Runaway Process
CPU utilisation is at 100% and something is consuming an entire core.
Container Cannot Reach Database
The database is healthy. The application container still cannot connect.
Broken DNS
Users report that the application is unavailable.
The web server responds correctly when accessed by IP address, but the hostname does not resolve.
Start here
ping app.example.com
dig app.example.com
dig @8.8.8.8 app.example.com
cat /etc/resolv.conf
ip route
Evidence
$ dig app.example.com
;; connection timed out; no servers could be reached
$ cat /etc/resolv.conf
nameserver 10.0.0.53
$ ping 10.0.10.20
64 bytes from 10.0.10.20: icmp_seq=1 ttl=64 time=0.4 ms
The application server can reach the network, but its configured DNS resolver at 10.0.0.53 is unavailable.
Fix
# Verify resolver reachability
ping 10.0.0.53
# Test another resolver
dig @1.1.1.1 app.example.com
# Correct the resolver configuration
# according to the environment's DNS design.
# Verify
dig app.example.com
"DNS is broken" is not a diagnosis. Determine whether the problem is the client, resolver, authoritative server, network path, record or delegation.
Disk at 100%
The application reports write failures.
write error: No space left on deviceStart here
df -h
df -i
du -xhd1 /
Evidence
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 50G 50G 0 100% /
$ du -xhd1 /
18G /var
12G /usr
8G /home
2G /opt
The numbers do not account for all of the used space.
Investigate further
lsof +L1
Evidence
$ lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NAME
java 2147 app 12w REG 8,2 8.5G 0 /var/log/app.log (deleted)
A process still has an open file descriptor to a deleted 8.5 GB log file.
Deleting the file removed the directory entry, but the storage remains allocated until the process closes the file descriptor.
Fix
# Identify the owning process
lsof +L1
# Restart or otherwise safely recycle the process
systemctl restart application.service
# Verify
df -h /
"du" shows files visible in the filesystem namespace. "df" shows filesystem allocation. When they disagree, deleted-but-open files are one of the first things to investigate.
Kubernetes CrashLoopBackOff
$ kubectl get pods
NAME READY STATUS
orders-api-7c8f9c7d8d 0/1 CrashLoopBackOff
Start here
kubectl describe pod orders-api-7c8f9c7d8d
kubectl logs orders-api-7c8f9c7d8d
kubectl logs orders-api-7c8f9c7d8d --previous
Evidence
$ kubectl logs orders-api-7c8f9c7d8d
ERROR: DATABASE_URL is not defined
Application startup failed
The container starts correctly, but the application exits because the required database configuration is missing.
Investigate the manifest
kubectl get deployment orders-api -o yaml
The deployment references the application image but does not provide the required environment variable or secret.
Fix
kubectl create secret generic orders-db \
--from-literal=DATABASE_URL='postgresql://...'
# Then reference the secret from the Deployment
# using env.valueFrom.secretKeyRef.
kubectl rollout restart deployment orders-api
kubectl get pods
CrashLoopBackOff is not the root cause. It is Kubernetes telling you that the container repeatedly starts and terminates. Always inspect the container logs and previous container logs.
Broken systemd Service
$ systemctl status api.service
Active: failed
Start here
systemctl status api.service
journalctl -u api.service -n 100
systemctl cat api.service
Evidence
$ systemctl status api.service
Process: 1821 ExecStart=/opt/api/start.sh
(code=exited, status=203/EXEC)
systemd cannot execute the configured
/opt/api/start.sh.
The script exists, but its executable permission was removed.
Verify
ls -l /opt/api/start.sh
-rw-r--r-- 1 root root 412 start.sh
Fix
chmod 755 /opt/api/start.sh
systemctl daemon-reload
systemctl restart api.service
systemctl status api.service
systemd errors often tell you exactly where to look. Read the exit status before changing random configuration.
TCP Connection Failure
The application reports:
connection refused: database:5432
Start here
ping database
nc -vz database 5432
ss -lntp
ip route
Evidence
$ nc -vz database 5432
Connection refused
$ ping database
64 bytes from 10.0.20.15
$ ss -lntp
LISTEN 0 128 127.0.0.1:5432
PostgreSQL is listening only on the loopback interface. The network path works, but the service is not accepting remote connections.
Fix
# PostgreSQL configuration must listen
# on the required interface.
listen_addresses = '*'
# Then restart PostgreSQL
systemctl restart postgresql
# Verify
ss -lntp | grep 5432
Connection refused is different from connection timeout. A refusal usually means the destination was reachable but nothing accepted the connection on that address/port.
Terraform Drift
The infrastructure repository has not changed, but Terraform reports changes during planning.
terraform plan
Evidence
~ aws_instance.web
instance_type = "t3.small" -> "t3.medium"
Investigate
terraform state show aws_instance.web
aws ec2 describe-instances ...
Someone changed the instance type directly in the cloud console. The real infrastructure no longer matches the declared configuration.
Decision
There are two valid directions.
- If the change was intentional, update the Terraform code and review it through version control.
- If the change was accidental, allow Terraform to reconcile the infrastructure back to the declared state.
Fix
# Review the proposed change
terraform plan
# Apply only after verifying the desired state
terraform apply
Terraform is not merely a provisioning tool. The value comes from maintaining a reliable relationship between declared state and real infrastructure.
Kubernetes Node NotReady
$ kubectl get nodes
NAME STATUS
worker-01 Ready
worker-02 NotReady
worker-03 Ready
Pods scheduled on worker-02 are failing.
Start here
kubectl describe node worker-02
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp
Evidence
Conditions:
Ready False
MemoryPressure False
DiskPressure True
Reason:
KubeletHasDiskPressure
Move to the node
df -h
df -i
du -xhd1 /var
journalctl -u kubelet
The node's filesystem used by the container runtime has reached the configured disk pressure threshold.
Fix
# Identify container/image/log consumption
du -xhd1 /var/lib/containerd
du -xhd1 /var/log
# Remove unnecessary data using the runtime's
# supported garbage-collection mechanisms.
# Verify
df -h
kubectl get node worker-02
A Kubernetes problem can still be a Linux problem. Kubernetes does not remove the operating system underneath it.
Certificate Expired
Users suddenly receive a browser certificate warning.
NET::ERR_CERT_DATE_INVALID
Start here
openssl s_client \
-connect app.example.com:443 \
-servername app.example.com
date
Evidence
notBefore=Jun 01 00:00:00 2026 GMT
notAfter=Sep 01 23:59:59 2026 GMT
The certificate expired and was not renewed.
Investigate the renewal path
certbot certificates
systemctl list-timers
journalctl -u certbot
Fix
# Renew using the environment's configured
# ACME / Certbot method.
certbot renew
# Verify
openssl s_client \
-connect app.example.com:443 \
-servername app.example.com
Certificate expiry is rarely a certificate problem. It is usually a lifecycle automation problem.
Runaway Process
The server is slow. CPU utilisation is close to 100%.
Start here
uptime
top
ps aux --sort=-%cpu | head
pidstat 1
Evidence
PID USER %CPU COMMAND
4812 app 399.8 /opt/app/worker
Investigate
ps -fp 4812
ls -l /proc/4812/exe
cat /proc/4812/cmdline
strace -p 4812
The worker has entered a tight retry loop after losing its downstream dependency.
Fix
# Correct the underlying application/configuration issue.
# Then safely restart the worker.
systemctl restart application-worker.service
# Verify CPU
top
High CPU is a symptom. Killing the process may restore capacity temporarily, but it does not explain why the process consumed the CPU.
Container Cannot Reach Database
The database is healthy. The application container cannot connect.
connection timed out: postgres:5432
Start inside the container
getent hosts postgres
nc -vz postgres 5432
ip addr
ip route
Evidence
$ getent hosts postgres
10.20.0.15 postgres
$ nc -vz postgres 5432
Connection timed out
Check the destination
ss -lntp | grep 5432
LISTEN 0 128 127.0.0.1:5432
The database hostname resolves correctly and the network path exists, but PostgreSQL is listening only on localhost.
Fix
# Configure PostgreSQL to listen on the
# required network interface.
listen_addresses = '*'
# Configure pg_hba.conf appropriately.
# Restart PostgreSQL.
systemctl restart postgresql
# Verify from the application container.
nc -vz postgres 5432
Separate the layers:
DNS → ROUTING → FIREWALL → LISTENER → APPLICATION
Test each layer instead of assuming "the database is down."
Start at the layer.
| Layer | Question | Useful tools |
|---|---|---|
| Process | Is the process running? | ps · top · systemctl |
| Memory | Is the system under memory pressure? | free · vmstat · top |
| Storage | Is space or inode capacity exhausted? | df · du · lsof |
| Network | Can the destination be reached? | ping · ip · ss · nc · traceroute |
| DNS | Does the name resolve correctly? | dig · host · getent |
| TLS | Is the certificate and handshake valid? | openssl · curl |
| Container | Does the workload start and remain healthy? | docker · crictl · logs |
| Kubernetes | Is desired state converging? | kubectl · events · logs |
| IaC | Does declared state match reality? | terraform plan · state |
Don't fix the symptom. Find the broken assumption.
↓
SYMPTOM
↓
OBSERVATION
↓
EVIDENCE
↓
HYPOTHESIS
↓
TEST
↓
ROOT CAUSE
↓
FIX
↓
VERIFY
↓
PREVENT
Production rewards engineers who follow the evidence.
The commands change. The platforms change. The infrastructure changes.
The troubleshooting process remains remarkably consistent.
OBSERVE.
MEASURE.
QUESTION.
TEST.
FIX.
VERIFY.