Most engineering organizations have a metrics problem disguised as an observability program. Dashboards multiply. Alert rules accumulate. The on-call rotation fires at 3am for things that do not matter, and stays quiet during incidents that do. By the time something is clearly broken, the signal is buried under noise.
Google’s Site Reliability Engineering team published a framework in their SRE book that cuts through this. They called it the four golden signals, and the premise is simple: for any service you want to monitor, four measurements tell you whether that service is healthy or sick. If you measure nothing else, measure these.
The framework has held up well since its publication. It applies equally to microservices and monoliths, to APIs and batch jobs, to managed cloud services and self-hosted infrastructure. It is not a tool recommendation or a vendor pitch. It is an observation about what matters.
The four signals are latency, traffic, errors, and saturation. Each one is worth understanding in depth.
Latency: How Long Things Take
Latency is the time it takes to service a request, from the moment the request arrives to the moment the response is sent. On the surface, this sounds simple. In practice, there are two mistakes teams consistently make with it.
The first mistake is measuring averages. Average latency is almost always the wrong metric. Consider a service where ninety-five percent of requests complete in 50 milliseconds, and five percent take 2,000 milliseconds. The average might look like 150 milliseconds — acceptable by most standards. The five percent of users experiencing two-second delays may represent your highest-value customers, your most complex operations, or a systematic problem that will eventually spread. The average hides all of it.
The right approach is percentile-based measurement. P50 (median) tells you what the typical user experiences. P95 tells you what your slower users experience. P99 and P99.9 tell you what your worst-case users experience. For customer-facing services, p99 is usually the number that belongs in your SLO. For internal services, p95 is often sufficient. The specific thresholds matter less than the discipline of looking at the tail rather than the middle.
Average latency is almost always the wrong metric. The average hides the users who matter most.
The second mistake is treating error latency the same as success latency. A request that fails in 2 milliseconds is fast, but it is not a successful fast request. It is a fast failure. When you mix error responses into your latency calculations, you artificially deflate the numbers, because errors often complete quickly. Track latency for successful requests and failed requests separately.
Traffic: The Demand on Your System
Traffic measures how much work your system is being asked to do. For an HTTP service, that is requests per second. For a message queue consumer, it is messages processed per minute. For a database, it is queries per second. The right unit depends on what your service does.
Traffic is the context that makes every other signal interpretable. A latency spike at 10 requests per second and a latency spike at 10,000 requests per second mean completely different things. The first is almost certainly a code problem. The second might be load-driven saturation. Without traffic as a baseline, you are pattern-matching without the most important variable.
Traffic also drives capacity planning. When you know your normal traffic patterns, you can identify anomalies (sudden drops often indicate upstream failures, not just low demand), project growth, and reason about what headroom you have before saturation becomes a concern. Teams that do not track traffic systematically tend to discover their capacity limits at the worst possible time.
For engineering leaders, traffic metrics are one of the cleaner bridges between technical health and business context. Request volume correlates with user activity. Drops in traffic during business hours are worth investigating. Spikes outside normal patterns may indicate abuse, misconfiguration, or unexpected success. Traffic belongs in your operational vocabulary.
Errors: The Rate of Failure
Errors measure the rate of requests that fail — not the count. This distinction is more important than it might appear.
Ten errors per minute in a service handling 100 requests per minute is a ten percent error rate. That is a serious problem. Ten errors per minute in a service handling 100,000 requests per minute is a rounding error, and may well be within your SLO budget. The raw number tells you almost nothing without the denominator.
Errors also come in two varieties that require different approaches. Explicit errors are the straightforward kind: HTTP 5xx responses, exceptions that bubble up to the caller, failed database transactions. Your infrastructure and frameworks usually surface these automatically. Implicit errors are trickier. An HTTP 200 with an empty body when data should be present. A response with stale data because a cache was not invalidated. A degraded feature silently returning default values instead of real ones. These do not appear in your error rate without deliberate instrumentation.
Error rate without traffic context is almost meaningless. Ten errors looks the same whether your system handles a hundred requests or a million.
When defining error SLOs, be specific about what counts as an error. A 502 from a gateway is an error. A 400 from a client sending a malformed request may not be — that is the client’s fault, not yours. A 404 may or may not be an error depending on whether it represents expected behavior. Being deliberate about these definitions prevents both false alarms and genuine problems being waved away.
Saturation: How Full the System Is
Saturation is the signal that predicts failure before it arrives. It measures how close your system is to its limits — the utilization of constrained resources like CPU, memory, disk I/O, thread pools, and connection pools.
Unlike the other three signals, saturation is most useful as a leading indicator. Latency, traffic, and errors describe what is happening right now. Saturation describes what is about to happen. A service running at 90 percent CPU utilization will handle the current load. Add a 15 percent traffic spike and it will not.
Different services have different saturation points. For stateless compute services, CPU and memory utilization are typically the relevant metrics. For databases, disk I/O and connection pool exhaustion matter more. For message queue consumers, queue depth is the clearest saturation signal — a queue that is growing faster than it is being consumed will eventually overwhelm the consumer regardless of how healthy the consumer looks in isolation.
Saturation also surfaces subtle problems that other signals miss. A thread pool that is constantly near capacity will handle individual requests quickly while creating queuing delays that only appear under load. A memory leak will show up in saturation metrics long before it causes an out-of-memory failure. A disk that is 95 percent full is not causing errors today, but it will at 100 percent, often in ways that are difficult to recover from quickly.
The practical implication is that saturation metrics belong in every runbook and every pre-incident checklist. Before a major release, a traffic event, or any significant operational change, check your saturation baselines. Knowing where your headroom is before you need it is far more useful than discovering you have none.
How the Signals Relate
The four golden signals are not independent. They form a causal chain that, once understood, makes incident diagnosis significantly faster.
Latency
Rising latency is often a downstream symptom. When you see it, look upstream at saturation and errors. A saturated dependency inflates latency. Retries on errors inflate latency. The fix is usually not in the latency measurement itself.
Traffic
Traffic changes are the most common driver of saturation and latency problems. An unexpected traffic increase should immediately prompt you to check saturation headroom. A drop should prompt an investigation into whether something upstream is broken.
Errors
Error rate spikes often accompany saturation events and are frequently caused by timeouts or cascading failures. They also directly inflate latency, because retries and circuit-breaker delays add time to the critical path.
Saturation
Saturation is the earliest warning signal in the chain. It typically rises before errors spike, and errors spike before latency becomes obviously degraded to users. Watch saturation first and you can often intervene before users notice anything.
A common incident pattern looks like this: traffic increases, saturation climbs, a resource limit is reached, errors begin, latency inflates as retries accumulate, and users report the service as slow or broken. Teams that see the saturation climb early and respond have a much shorter incident. Teams that only notice when latency is degraded are already in the middle of it.
From Signals to SLOs
The golden signals become genuinely powerful when you use them to define Service Level Objectives. An SLO is a reliability target expressed as a measurable threshold over a time window. The golden signals give you a natural vocabulary for writing them.
A latency SLO might read: 99 percent of requests will complete in under 300 milliseconds, measured over a 30-day rolling window. An error rate SLO might read: fewer than 0.1 percent of requests will result in a 5xx error over a 7-day window. A saturation SLO might be expressed as: CPU utilization will remain below 80 percent in steady state.
These targets are meaningful because they are measurable, they connect to user experience, and they provide a clear signal for when to invest in reliability work. When you consistently meet your SLOs, you have reliability headroom to spend on feature development. When you miss them, you have a clear mandate for operational investment.
Setting SLOs for the first time is an exercise in honest assessment rather than aspiration. Start by measuring where you actually are, not where you think you should be. A team that is currently at 99.5 percent availability should set its initial SLO at 99.5 or slightly below, then work toward tighter targets over time. An aspirational SLO that you never meet is worse than a realistic one that you consistently hit.
Common Mistakes
Teams adopting the golden signals framework for the first time tend to encounter the same problems.
Alerting on Averages
Average latency alerts miss the users having the worst experience. Set alert thresholds at p95 or p99. The goal is to catch problems that affect real users, not to produce clean-looking dashboards.
Alerting on Error Counts Instead of Rates
An error count threshold that makes sense at low traffic triggers constantly at high traffic, or misses genuine problems at low traffic. Always alert on error rate as a percentage of total traffic.
Ignoring Implicit Errors
HTTP 200 responses with degraded or missing content are errors even when the status code says otherwise. Define what a successful response actually means for each service and instrument accordingly.
Treating Saturation as an Afterthought
Saturation is the only signal that warns you before a problem fully materializes. Teams that only instrument the reactive signals — latency and errors — lose the opportunity to intervene early. Instrument saturation first.
Alerting on Everything Instead of Signals
The value of the golden signals framework is its constraint. Teams that add dozens of custom alerts on top of it re-create the noise problem they were trying to solve. Reserve on-call alerts for signal violations. Use dashboards for everything else.
Getting Started Without Starting Over
If your organization has existing monitoring that does not align with this framework, you do not need to rip it out and rebuild. Start by identifying which existing metrics map to each signal. You likely have something for latency and errors already. Traffic and saturation may require new instrumentation.
Establish baselines before you set SLOs. Run your current metrics for four to six weeks, understand your normal patterns, identify seasonal variation, and then set targets that reflect reality. An SLO you cannot currently meet is a target, not a commitment. Mark it as such until you have the reliability work to back it up.
Prioritize instrumentation on your most critical services first. Not the most complex services, not the most recently built, but the ones where failure has the greatest impact on users or business outcomes. The golden signals are most valuable where reliability matters most.
Once you have a working set of signal-based dashboards and alerts on your critical services, the framework scales naturally. New services instrument against the same four categories. On-call engineers develop a shared vocabulary. Incident postmortems have a common analytical starting point. The overhead of building all of this from scratch is real, but it pays back quickly in reduced incident duration and better operational clarity.
How VergeOps Can Help
VergeOps helps engineering organizations build observability programs grounded in the golden signals framework, from initial instrumentation through SLO definition and alerting architecture.
Observability Assessment. We evaluate your current monitoring and alerting against the golden signals framework, identify the gaps that are creating noise or blind spots, and provide a prioritized roadmap for improvement.
SLO Definition and Rollout. We facilitate the process of defining meaningful SLOs for your critical services — from agreeing on what counts as an error to setting realistic targets that drive the right engineering investments.
Alerting Architecture. We help design and implement alerting strategies that reduce noise, preserve on-call sanity, and ensure your team is paged on signal violations before users report problems.
Platform Engineering Support. For organizations building or maturing internal developer platforms, we provide architectural guidance on embedding observability into the platform so golden signal instrumentation is the default rather than the exception.
Contact us to discuss how your organization can move from dashboard sprawl to operational clarity.