In this analysis
The service is up and the task is broken
A customer can open the site, authenticate, and submit a request. The final confirmation never arrives. Infrastructure monitoring reports healthy servers and an available homepage. From the customer’s perspective, the service has failed. This hypothetical gap between component health and completed work is why reliability should begin with a user journey rather than a machine inventory.
Google’s SRE guidance on service-level objectives connects reliability targets to indicators that reflect service behavior. Its monitoring chapter distinguishes externally visible symptoms from internal causes. 12 The practical implication is to define the event that counts as success. A successful HTTP response may be a useful signal, but it is not necessarily the same as a completed booking, saved document, or delivered result.
Define the promise before drawing the chart
For a hypothetical analysis platform, a meaningful journey might start when a user submits a valid dataset and end when the resulting report can be opened. The indicator must define eligible requests, time limits, and exclusions. Otherwise, a team can improve the apparent success rate by moving inconvenient failures outside the denominator.
We recommend distinguishing interactive steps from asynchronous work. An accepted job and a completed job are different states. Measure both when both matter, and make their relationship visible. A target should be demanding enough to reflect user needs and realistic enough to guide trade-offs. It should not be chosen solely because the current system happens to meet it.
Follow the request across boundaries
OpenTelemetry describes traces, metrics, and logs as distinct telemetry signals. Dapper documents a distributed tracing system designed to follow requests across large-scale services. 59 These contributions help explain why an aggregate dashboard cannot always identify where a particular user journey stalled.
A trace can connect a slow report to a storage call, queue delay, or downstream dependency. Metrics reveal whether the problem is isolated or widespread. Logs can provide event details, subject to appropriate handling of sensitive data. None should become a reason to collect every payload indiscriminately. Instrument the path needed to investigate the service promise and retain enough context to connect symptoms with plausible causes.
The tail is where some users live
Dean and Barroso’s The Tail at Scale examines how latency variability can affect large distributed services. 10 The relevance is not limited to extremely large systems: an average response time can hide a minority of requests that take much longer. A journey that waits on several dependencies can be particularly sensitive to a slow component.
Report latency distributions for important workflows and inspect the long-running cases. Separate ordinary processing variation from timeouts, retries, and queueing. Do not assume that improving the mean will improve every user’s experience. For a service with a deadline, the fraction completed within that deadline may be more informative than an average that mixes very fast successes with a smaller number of unusably slow ones.
Retries can repair a failure or amplify it
AWS’s guidance explains how timeouts, retries, backoff, and jitter interact with overloaded dependencies. Its discussion of idempotent APIs addresses the uncertainty of repeated operations. 78 A retry is therefore a controlled intervention, not a harmless default.
Imagine the analysis platform retrying the same expensive job whenever a response is delayed. Without deduplication and a bounded policy, a temporary slowdown can create more work and worsen the slowdown. Define which failures are retryable, set a budget, and preserve the identity of the intended operation. A retry policy should also account for whether the user has already abandoned the task. Reliability engineering includes knowing when additional attempts stop helping.
Alert on a decision someone can make
Google’s SRE Workbook discusses alerting on SLOs, including approaches that relate alerts to the rate at which an error budget is consumed. Prometheus provides alerting rules with mechanisms for expressing conditions over time. 36 The objective is not to produce more notifications; it is to identify conditions that warrant timely action.
A useful alert names the affected service promise, provides relevant context, and connects to a response the operator can take. A transient internal fluctuation may belong on a dashboard rather than in a page. Conversely, a small but sustained failure rate on a critical journey may deserve attention even when every server appears healthy. Review whether alerts detect user harm early enough to change the outcome.
Recovery includes learning
Google’s guidance on postmortem culture emphasizes learning from incidents and documenting what happened without reducing the analysis to individual blame. 4 An incident can reveal a mismatch between the system’s design assumptions and its actual operating conditions.
For the analysis platform, a report-generation failure might expose an unbounded queue, an unclear ownership boundary, or a missing way for users to recover their work. The postmortem should connect contributing conditions to specific improvements and accountable follow-up. “Be more careful” is not a durable repair. Neither is a list of actions so broad that none can be completed. Preserve the evidence that explains why the proposed changes address the failure.
The denominator makes the objective meaningful
An availability indicator can be expressed as successful eligible events divided by all eligible events. Every word matters. If a report is accepted but never completed, counting only the acceptance response can make the service appear healthy. If invalid requests are excluded, the exclusion rule must be defined consistently rather than adjusted after an incident.
For time-bounded journeys, success can include both correct completion and completion before a specified deadline. Google’s SLO guidance provides the context for selecting meaningful indicators and objectives. 1 We recommend publishing the event definition internally alongside the target. Then review edge cases: cancelled jobs, duplicate submissions, partial results, and dependency failures. An objective becomes actionable when operators can agree whether a particular event met it, without renegotiating the meaning of success while the system is failing.
Reliability is a product property
We propose a recurring review organized around important user journeys: whether the promise was met, which failures consumed the most user time, and which engineering changes would remove recurring causes. Keep component metrics, but relate them to the work people are trying to finish.
The literature provides methods for objectives, telemetry, latency analysis, and recovery. It does not offer a universal availability percentage that makes every product reliable. Our conclusion is that a service deserves the label when people can complete its important tasks within understood limits and recover when those limits are exceeded. A green infrastructure dashboard is valuable evidence. It is only one part of that argument.
Measure the completed user task, then connect the measure to a concrete operational response.
Sources & methodology
This editorial synthesizes primary research, technical standards, and original engineering documentation. It is not an original experimental study or a systematic literature review. Research findings apply to the conditions studied; product recommendations are our interpretation. Figures explain concepts and do not present experimental measurements.
Sources checked on 31 October 2026. Publication dates and preprint versions may differ; the linked record identifies the referenced work. To suggest a correction, email contact@data-sorcerers.com.
- Google SRE — Service Level Objectives ↗Back to text ↑
- Ewaschuk / Google SRE — Monitoring Distributed Systems ↗Back to text ↑
- Google SRE Workbook — Alerting on SLOs ↗Back to text ↑
- Google SRE — Postmortem Culture: Learning from Failure ↗Back to text ↑
- OpenTelemetry — Signals ↗Back to text ↑
- Prometheus — Alerting Rules ↗Back to text ↑
- AWS Builders’ Library — Timeouts, Retries, and Backoff with Jitter ↗Back to text ↑
- Featonby / AWS Builders’ Library — Making Retries Safe with Idempotent APIs ↗Back to text ↑
- Sigelman et al. (2010) — Dapper, a Large-Scale Distributed Systems Tracing Infrastructure ↗Back to text ↑
- Dean and Barroso (2013) — The Tail at Scale ↗Back to text ↑
