The reliability problem nobody warned you about
When your customers call you at 2 AM because their production system is down, they don’t care about your architecture. They care about one thing: when will it be back? In fifteen years of delivering technology solutions across industries — from financial services to manufacturing to healthcare — I’ve seen brilliant engineering teams build sophisticated systems that still collapse under pressure because they confused building things with operating things.
Site Reliability Engineering is the discipline that bridges that gap. And when your estate lives on AWS, the tooling available to you today makes what used to require entire platform teams achievable with a leaner, more focused SRE practice. But tooling alone doesn’t make you reliable. Mindset, process, and architecture must align. This article is about all three.
Let me walk you through how we architect, instrument, and operate reliable systems on AWS — from the principles down to the specific services, with the real trade-offs included.
SRE isn’t DevOps with a fancier name
The confusion between SRE and DevOps is worth clearing up before we go deeper. DevOps is a cultural and organisational philosophy: break down silos, automate delivery pipelines, and make developers responsible for what they ship. SRE is a specific implementation of operational excellence, originally coined at Google, that applies software engineering principles to infrastructure and operations problems.
The practical difference: DevOps tells you how to collaborate. SRE tells you what to measure and how to decide. The core SRE contract is built on three constructs that every engineering leader on AWS must internalise.
Service Level Indicators (SLIs) are the raw metrics that represent user experience — request latency at the 99th percentile, error rate per minute, availability percentage over a rolling window. An SLI without a target is just a dashboard nobody acts on.
Service Level Objectives (SLOs) are the targets you set against your SLIs — the internal contract you make with yourself about what “good” looks like. 99.9% availability over a 30-day rolling window. P99 latency under 400ms for API calls. The SLO is where engineering discipline meets business consequence.
Error budgets are the mechanism that makes SRE self-regulating. If your SLO is 99.9% availability, you have 43.8 minutes of allowable downtime per month. That budget belongs to the engineering team. Spend it on incidents? You’re on reliability lockdown — no new features until the budget recovers. Have budget to spare? Ship faster, experiment bolder. Error budgets turn the abstract tension between reliability and velocity into a concrete, negotiable resource.
This framework is the operating system for everything that follows.
The AWS-native SRE architecture
Here is how we architect an SRE capability on AWS. This is not theoretical — it’s the reference architecture we deploy with clients across our managed services practice.
Each layer in this architecture has a distinct SRE responsibility. The user traffic layer (Route 53, CloudFront, WAF) is your first line of defence — health checks here aren’t just ping tests, they’re the earliest signal that something in your stack is degrading. The application layer is where most SLIs are born. The data layer is where most SLOs are broken.
The observability platform sits orthogonally to all three layers — it reads from everything, speaks to the SLO engine, and feeds the incident response machinery. This is the architectural decision most teams get wrong: they instrument the application layer and forget that a slow Aurora query or a DynamoDB hot partition is what’s actually causing the SLI to miss.
Building SLIs that actually reflect user experience
The most common SRE mistake I see in the field is instrumenting what’s easy to measure rather than what users actually experience. CPU utilisation is easy to measure. Whether your checkout flow completed in under two seconds is what the user cares about.
On AWS, you have four primary mechanisms for capturing user-centric SLIs.
CloudWatch Metric Math lets you compute composite metrics that directly express your SLI. A basic availability SLI for an API might look like this:
SLI = (RequestCount - 5xxErrorCount) / RequestCount × 100
You define this as a CloudWatch metric math expression, add an alarm at your SLO threshold, and you have a live SLI. The sophistication comes in windowing — a 1-minute alarm tells you something is wrong. A 30-day rolling window tells you whether you’re burning through your error budget.
CloudWatch Synthetics is underused and undervalued. Rather than measuring what your systems report about themselves, Synthetics runs real browser-based and API-based canaries from multiple AWS regions against your production endpoints. The difference between these two approaches is the difference between asking a patient how they feel and actually running the blood test. When your ALB shows zero 5xx errors but your Synthetics canary is failing, you have a routing, DNS, or CDN issue that internal metrics would never catch.
X-Ray service maps give you distributed trace visibility across your entire service graph. For SRE, the critical value of X-Ray is latency attribution — when your P99 latency breaches its SLO, X-Ray tells you exactly which service in the call chain is the culprit. Without this, you’re triaging a system-wide incident by reading individual service logs, which is the investigative equivalent of reading a novel backwards.
CloudWatch Container Insights is essential if you’re running EKS or ECS. Pod-level and node-level metrics, combined with application-level metrics, give you the full picture when a Kubernetes node is under memory pressure and your SLIs are starting to slip.
Error budget mechanics on AWS
Defining an error budget is straightforward. Enforcing it as an operational policy requires some engineering. Here’s how we build error budget tracking on AWS in practice.
The core mechanism is a Lambda function that runs on a schedule (every 15 minutes) and does three things: queries CloudWatch for your SLI metric over the current billing/rolling period, computes remaining error budget as (1 - SLO_target) × period_minutes - burned_minutes, and writes the result back to a custom CloudWatch metric namespace alongside a DynamoDB record for historical trending.
The CloudWatch dashboard then surfaces this as a gauge: green when budget > 50%, amber when 25–50%, red when below 25%, and a separate alarm that fires into your incident channel when you’ve consumed 100% of the budget — which triggers the feature freeze protocol.
The feature freeze protocol is the part most teams skip because it requires organisational courage. When error budget is exhausted, the SRE team has the authority to pause non-reliability feature deployments. Not suggest. Pause. This requires executive sponsorship and a written policy that predates any incident. If you build the tooling without the policy, you have a dashboard nobody acts on.
Chaos engineering on AWS: breaking things intentionally
Reliability isn’t proven in steady state. It’s proven under failure. Chaos engineering — deliberately injecting faults into production or production-equivalent environments — is how SRE teams build confidence in their systems’ blast radius.
AWS Fault Injection Service (AWS FIS) is the native chaos engineering tool, and it’s matured significantly. The experiments that deliver the most reliability insight in our practice are: EC2 instance termination to validate auto-scaling recovery time, RDS failover injection to measure Aurora cluster promotion latency against your SLO, API throttling injection to validate your retry and circuit-breaker logic, and AZ blackout simulation to verify multi-AZ failover actually works the way your runbook says it does.
The last one is the most revealing. I have lost count of how many architectures look multi-AZ on the diagram but have a single-AZ dependency buried in the data layer — a self-managed Redis node, an NFS mount, a legacy database connection string hardcoded to a specific subnet. FIS AZ blackout finds these in the pre-production environment rather than during a real event.
The SRE operational model for managed services
For our managed services practice, SRE is not a bolt-on. It’s the operating model. Every workload we onboard goes through an SRE readiness assessment that covers five dimensions: SLI definition (do we have user-centric metrics?), SLO agreement (has the client signed off on the target?), alerting coverage (are all SLI-affecting failure modes instrumented?), runbook completeness (can an on-call engineer resolve the top 10 incident types without escalating?), and blast radius mapping (do we know what fails when each dependency fails?).
Workloads that don’t pass the readiness assessment don’t go into managed operations. This sounds strict. In practice, it protects the client and the delivery team from a relationship where both parties are flying blind.
The discipline of SRE, applied consistently through AWS-native tooling, transforms operations from a reactive cost centre into a proactive engineering function. That transformation is what separates a managed services practice that wins renewals from one that fights them.
