Why Averages Lie About Latency
Imagine an API that answers 99 requests in 100 ms and one request in 10 seconds. Its average response time is about 199 ms — a number that describes precisely none of the hundred requests. The 99 fast ones were twice as fast as the average suggests; the slow one was fifty times worse. This is not a contrived example: real-world latency distributions are heavily skewed, with a long tail of slow requests caused by garbage collection pauses, cold caches, lock contention, and overloaded dependencies.
Averages hide that tail by design — they smear the outliers across all the fast requests until everything looks fine. A latency chart built on averages can stay perfectly flat while a growing fraction of your users stares at timeouts. That is why every serious monitoring practice replaces (or at least supplements) the average with percentiles, which answer a much more honest question: "how slow is it for the unluckiest X% of my requests?"
What p50, p95, and p99 Actually Mean
A percentile is a simple idea: sort all response times from fastest to slowest, then read the value at a given position. p50 (the median) is the middle value — half your requests were faster, half were slower. It describes the typical experience and is far more robust than the average, because a handful of extreme outliers cannot drag it around.
p95 is the value below which 95% of requests fall — put differently, 1 in 20 requests is slower than this. It captures the experience of your unlucky-but-not-rare users. p99 marks the slowest 1% — 1 in 100 requests. It sounds negligible until you do the arithmetic: at one million requests per day, p99 describes 10,000 requests, every single day.
A healthy way to read the trio: p50 tells you what your API feels like, p95 tells you what your users complain about, and p99 tells you what your systems are hiding from you. The gap between them matters as much as the values: a p50 of 100 ms with a p99 of 150 ms is a tight, predictable service; a p50 of 100 ms with a p99 of 4 seconds is a time bomb.
Why Tail Latency Hits Your Best Customers Hardest
The counterintuitive property of tail latency is that it does not distribute evenly — it concentrates on your most active users. A user who makes one request has a 1% chance of hitting your p99. A user whose page load fans out into 40 API calls has a 33% chance that at least one of them lands in the slowest 1%. And because the page is only as fast as its slowest call, your heaviest users — the dashboards, the integrations, the power customers — experience your p99 latency routinely, not exceptionally.
This fan-out effect is why large engineering organizations obsess over the tail. In microservice architectures the problem compounds: if service A calls B, which calls C, one slow hop stalls the whole chain, holds a worker thread hostage, and under load can ripple into a cascading slowdown of services that were themselves perfectly healthy. Tail latency is not a cosmetic metric — it is the leading indicator of the saturation problems that later become outages.
Which Percentile Should You Alert On?
Alerting on the average is worse than useless — it creates false confidence. But alerting on the wrong percentile creates noise instead. A practical scheme:
Alert on p95 or p99 for user-facing endpoints. Set the threshold from your own baseline, not from folklore: measure a normal week, then alert when the percentile sustainably exceeds, say, 2–3× its baseline. A brief p99 spike during a deploy is normal; a p99 that stays elevated for ten minutes is a real degradation. Watch p50 for drift, not for alerts. A slowly rising median is the signature of growing datasets, missing indexes, or creeping payload sizes — review it weekly rather than paging on it. Track the p50/p99 gap. A widening gap with a stable median means the tail is degrading first — the classic early warning of an overloaded dependency or connection pool.
And measure where it matters: percentiles computed from your server logs miss DNS, TLS, and network time entirely. External synthetic checks from multiple regions measure the latency your clients actually experience, which is the number your users judge you by.
Common Percentile Pitfalls
Three mistakes show up in almost every latency dashboard. Averaging percentiles: the average of five per-minute p99 values is not the p99 of the five-minute window — percentiles do not compose. When aggregating across time windows or across regions, compute the percentile over the merged data (or use a mergeable sketch), never an average of percentiles. Ignoring sample size: a p99 computed from 50 requests is essentially one unlucky request masquerading as a trend; give low-traffic endpoints wider windows before trusting their tail. Comparing percentiles across different mixes: a p95 that "improved" after a traffic drop may simply reflect that the heavy queries left — segment by endpoint and request type before celebrating.
None of this requires sophisticated tooling to get right — it requires measuring continuously, from outside, with thresholds per monitor. A latency threshold on a synthetic check is the simplest honest implementation: every check either met your latency budget or it did not, and a run of failures is a degradation you hear about while it is still a performance problem rather than an outage.
