DESIGN FOR FAILURE.
ASSUME EVERYTHING BREAKS.
Architecture is not about drawing boxes. It is about understanding dependencies, failure domains, traffic, state, recovery and operational reality.
Start with the failure model.
A production architecture should answer one question before technology is selected:
Not whether it can fail. It will.
- What is the failure domain?
- What happens to traffic?
- Where does state live?
- Can the system recover automatically?
- What requires human intervention?
- How much data can be lost?
- How long can the service be unavailable?
- How do we know it has failed?
Don't build one giant blast radius.
Separate systems across hosts, racks, availability zones, networks and administrative boundaries where appropriate.
Know where your data lives.
Stateless applications are easy to replace. Stateful systems require deliberate storage, replication and recovery design.
Every dependency is part of your system.
DNS, databases, identity, certificates, storage, networks and external APIs all belong in the failure model.
HA is not DR.
High availability reduces downtime. Disaster recovery restores service after larger failures. They solve different problems.
A production service.
A simplified reference architecture for a modern internet-facing application.
INTERNET
|
v
+---------------+
| DNS |
+---------------+
|
v
+---------------+
| CDN / WAF |
+---------------+
|
v
+---------------+
| LOAD BALANCER |
+---------------+
|
+---------+---------+
| |
v v
+-----------+ +-----------+
| INGRESS | | INGRESS |
| AZ-1 | | AZ-2 |
+-----------+ +-----------+
| |
+---------+---------+
|
v
+---------------+
| APPLICATION |
| KUBERNETES |
+---------------+
|
+----------+----------+
| | |
v v v
+-------+ +-------+ +-------+
| REDIS | | API | | QUEUE |
+-------+ +-------+ +-------+
|
v
+-------------+
| DATABASE |
| PRIMARY / |
| REPLICA |
+-------------+
|
v
+-------------+
| BACKUPS |
+-------------+
The diagram is not the architecture. The relationships between the components are.
For every arrow ask:
What protocol?
What authentication?
What happens on timeout?
What happens on retry?
Separate the blast radius.
REGION
|
+-----------------+-----------------+
| |
v v
AVAILABILITY ZONE A AVAILABILITY ZONE B
| |
+----+----+ +----+----+
| | | |
v v v v
NODE-01 NODE-02 NODE-03 NODE-04
| | | |
v v v v
PODS PODS PODS PODS
A single server disappears. Workloads should be rescheduled elsewhere.
An entire availability zone becomes unavailable. Replicas must exist outside that zone.
The entire region is unavailable. Multi-region architecture and DR become relevant.
A deployment introduces a defect. Rollback and progressive delivery become critical.
Three replicas on three nodes in the same failure domain can still disappear together.
Remove single points of failure.
| Component | Single Instance | HA Approach |
|---|---|---|
| Load Balancer | Single endpoint | Managed / redundant pair |
| Ingress | One replica | Multiple replicas across nodes |
| Application | One pod | Multiple replicas |
| Node | Single worker | Multiple workers |
| Database | Single primary | Replication / managed HA |
| Storage | Single disk | Redundancy + backup |
| DNS | Single resolver | Redundant infrastructure |
Removing one single point of failure can expose another.
Example: three Kubernetes replicas do not provide application availability if all three depend on one unavailable database.
Treat the cluster as a distributed system.
KUBERNETES CLUSTER
+-----------------------+
| CONTROL PLANE |
| |
| API Server |
| Scheduler |
| Controllers |
| etcd |
+-----------+-----------+
|
+--------------+--------------+
| | |
v v v
+---------+ +---------+ +---------+
| NODE 01 | | NODE 02 | | NODE 03 |
| | | | | |
| kubelet | | kubelet | | kubelet |
| runtime | | runtime | | runtime |
| | | | | |
| PODS | | PODS | | PODS |
+---------+ +---------+ +---------+
Design questions
- What happens if a node disappears?
- What happens if the control plane is unavailable?
- Where are replicas scheduled?
- Can workloads accidentally concentrate on one node?
- Where does persistent state live?
- What happens when the cluster network fails?
- How are certificates rotated?
- How does a deployment get rolled back?
Spread workloads.
Prevent replicas from becoming dependent on the same node or failure domain.
Protect availability.
PodDisruptionBudgets can limit voluntary disruption during maintenance operations.
Health must be meaningful.
Liveness, readiness and startup probes have different purposes. Don't use them interchangeably.
Make scheduling predictable.
Requests and limits influence scheduling, capacity planning and workload behaviour.
Follow the packet.
Application architecture is inseparable from network architecture.
CLIENT | | HTTPS / 443 v DNS | v CDN / WAF | v LOAD BALANCER | | HTTP / HTTPS v INGRESS | | Service v APPLICATION | | TCP v DATABASE
Traffic entering or leaving the environment.
Traffic between internal services and workloads.
Separate systems according to trust boundaries and operational requirements.
Name resolution is infrastructure. When DNS fails, everything appears broken.
Don't start with the application. Start with the packet path.
DNS → Route → Firewall → Load Balancer → Listener → Ingress → Service → Pod → Application → Dependency
Use the cloud as infrastructure. Not as architecture.
Cloud services provide primitives. Architecture determines how those primitives interact.
CLOUD ACCOUNT
|
+--------------+--------------+
| |
v v
NETWORK IDENTITY
| |
+----+----+ +----+----+
| | | |
v v v v
PUBLIC PRIVATE IAM SECRETS
SUBNET SUBNET
| |
v v
LB WORKLOADS
|
v
DATABASE
|
v
BACKUPS
Architecture layers
| Layer | Questions |
|---|---|
| Identity | Who can access what? |
| Network | Who can communicate with whom? |
| Compute | Where does the workload run? |
| Storage | Where does persistent state live? |
| Security | What happens when trust is breached? |
| Observability | How do we know the system is healthy? |
| Recovery | How do we restore service? |
Make the desired state visible.
GIT REPOSITORY
|
| commit
v
+-----------+
| CI/CD |
+-----------+
|
v
+-----------+
| IMAGE |
| REGISTRY |
+-----------+
|
v
+-----------+
| GITOPS |
| CONTROLLER|
+-----------+
|
v
KUBERNETES API
|
v
CLUSTER
|
v
WORKLOADS
Git is the desired state.
Configuration should be reviewable, versioned and auditable.
Detect reality diverging.
Manual changes should become visible rather than quietly becoming the new production configuration.
Previous state should be recoverable.
A deployment is not complete if there is no reliable way to reverse it.
Know who changed what.
Git history provides an operational record of configuration changes.
You cannot operate what you cannot see.
What is happening?
CPU, memory, latency, throughput, errors, saturation and application-specific signals.
What happened?
Events, errors, requests, state transitions and diagnostic information.
Where did the request go?
Follow distributed requests across services and dependencies.
What requires action?
Alert on symptoms and service impact rather than every metric that changes.
LATENCY ERRORS TRAFFIC SATURATION
Observability should help an engineer move from symptom → evidence → cause → action.
Assume compromise.
Security architecture is not a firewall diagram. It is the collection of controls limiting what an attacker can do after something goes wrong.
INTERNET
|
[ WAF ]
|
[ LOAD BALANCER ]
|
[ INGRESS ]
|
+---------+---------+
| |
[ SERVICE A ] [ SERVICE B ]
| |
+---------+---------+
|
[ DATABASE ]
|
[ BACKUP ]
Every identity gets only the permissions required to perform its function.
Credentials should not be embedded in images, repositories or configuration files.
Limit lateral movement between systems and trust zones.
Authentication, authorization and administrative actions should leave evidence.
Stateless is easy. State is where architecture gets serious.
REQUEST | v +--------+ | POD A | +--------+ POD A dies | v +--------+ | POD B | +--------+
The workload can simply be replaced.
REQUEST | v +--------+ | APP | +--------+ | v +----------+ | DATABASE | +----------+ | +---- REPLICATION | +---- BACKUP | +---- RECOVERY
The workload and its data have different recovery requirements.
- What is the source of truth?
- How is data replicated?
- How is corruption detected?
- How often are backups taken?
- Have restores actually been tested?
- What is the RPO?
- What is the RTO?
RPO and RTO are architecture decisions.
| Term | Meaning | Example |
|---|---|---|
| RPO | Maximum acceptable data loss | 15 minutes |
| RTO | Maximum acceptable recovery time | 30 minutes |
| HA | Reduce normal service interruption | Multi-AZ application |
| DR | Recover from major failure | Secondary region |
Recovery procedures need to be tested.
Capacity is part of reliability.
Understand utilization, saturation and CPU throttling.
Memory pressure can produce eviction, OOM kills and cascading failure.
Disk exhaustion can break databases, logging, schedulers and operating systems.
Bandwidth, latency, packet loss and connection limits can become application bottlenecks.
AVAILABLE CAPACITY
|
v
CURRENT LOAD
|
v
GROWTH RATE
|
v
FAILURE RESERVE
|
v
SCALING DECISION
The architecture review checklist.
-
Traffic
How does a request enter and leave the system? -
Dependencies
What external and internal systems does it require? -
Failure
What happens when each component disappears? -
State
Where does persistent data live? -
Scale
What happens when traffic increases by 10×? -
Security
What happens if an identity or workload is compromised? -
Observability
How will the operations team know something is wrong? -
Deployment
How does new software reach production? -
Rollback
How quickly can a bad deployment be reversed? -
Recovery
How is the system restored after catastrophic failure? -
Cost
What does the architecture cost at current and peak scale? -
Human factors
Can an engineer actually operate this system at 03:00?
Would you want to operate this system at 03:00?
Production architecture should be evaluated by the person who has to troubleshoot it when everything is on fire.
If diagnosing a failure requires tribal knowledge, ten dashboards and three people who happen to be asleep, the architecture has an operational problem.
- Unknown dependencies
- Manual production changes
- No tested rollback
- No restore testing
- Single points of failure
- Missing monitoring
- Unbounded retries
- Unclear ownership
Good architecture makes failure boring.
A good system does not prevent every failure.
A good system limits the blast radius.
A good system detects failure quickly.
A good system recovers predictably.
A good system tells the engineer what happened.
A good system can be changed without fear.
Test the architecture.
Break systems deliberately and learn how they recover.
Learn from production.
Real incidents expose architectural assumptions.
Distributed systems at scale.
Explore the platform layer.
Systems first.
Understand the engineering philosophy behind ROOT // SYSTEMS.