Your average latency looks fine. Why are users still waiting?

One number, very different experiences

Imagine 100 requests: 95 finish in 100 milliseconds and five take two seconds. The mean is 195 milliseconds. That may look acceptable against a 200-millisecond target, even though one request in twenty is making someone wait two seconds. This is an illustrative example, not a measured Hotpath result.

The mean is still useful. It simply answers a different question from “How slow are the slow requests?” Look at the distribution alongside it. The median describes the middle observation; a high percentile describes a point toward the slower end. Percentiles are not maximums and do not, by themselves, explain the cause.

Google’s SRE guidance recommends examining percentiles because averages can conceal variation in latency. Use several views rather than turning p99 into another isolated headline. Reference: Google SRE, Monitoring.

Measure the journey your user experiences

Decide where the timer starts and ends. A server-side timer may exclude time waiting for a connection, network transfer, or rendering in the browser. A fast service can sit inside a slow end-to-end journey. Record the boundary in the dashboard title or accompanying documentation so readers do not mistake one measurement for another.

Separate operations that have different expectations. A background report and an interactive lookup should not share a single latency objective merely because they use the same backend. Distinguish successful requests from failures and timeouts: a quick error is not a fast, successful experience.

Make the slow requests inspectable

Choose a window containing a reported slowdown. Correlate the slow requests with traces, application events, and resource measurements in that same window. Useful questions include: Which operation was involved? Was the delay before useful work started? Did a dependency dominate the duration? Was the workload or payload different?

A request that waits on a queue needs a different investigation from one spending its time doing computation. A request that triggers a cache miss may follow a different path from most traffic. These are hypotheses to test, not diagnoses to infer from a percentile graph.

Compare like with like

Keep the operation mix, data size, concurrency, environment, and measurement boundary comparable when evaluating a change. Record the number of observations as well as the percentile: an extreme percentile calculated from a small sample offers limited insight into rare behavior.

Inspect the measurement tooling too. Confirm how it treats timeouts and how it combines observations across instances. A summary that looks precise may still represent a different population from the one you intended to measure.

Turn the finding into a regression check

Once an experiment identifies a cause and a change improves it, retain a representative test. Record both the benefit and the costs: perhaps memory use increased, a cache requires invalidation, or throughput changed. Keep correctness and recovery behavior in the review.

The useful outcome is a system your team can explain and improve. A better average is welcome; a clearer understanding of the requests that hurt users is what makes that progress repeatable.

A starting checklist

• Define the user operation and timing boundary.
• Inspect the distribution, request count, errors, and timeouts.
• Choose a slow window and follow representative requests.
• Test one evidence-backed hypothesis under comparable conditions.
• Document the tradeoffs and preserve a regression test.