Root cause in minutes: a method for distributed systems

In a distributed system, a single customer operation can pass through a mobile app, an API gateway, several microservices, a message queue, two databases and an external provider. When something fails, the symptom shows up in one place and the cause usually sits somewhere else.

Under pressure, teams tend to investigate what they know best. The database team checks the database, the network team checks the network, and each confirms its part is fine. Time goes into ruling things out without order. An explicit method does not replace experience, but it keeps triage from depending on who happens to be on call.

One clarification before starting: restoring service comes first. If there is a safe, obvious mitigation, such as rolling back a recent deployment, apply it, and continue the investigation once the service is stable.

Step 1: scope the impact

The first question is not "what broke?" but "what is failing, for whom and since when?". Scoping the impact narrows the search space more than any other action.

• Which operations: are all transactions failing or only some? If only card payments fail and logins work, a large part of the system is already ruled out.
• Which users: everyone or a segment? An app version, a channel, a country, a customer type.
• Where: one region, one availability zone, one cluster, one node?
• Since when: the exact start time is the most valuable input for step 3.
• How: errors, slowness, wrong results? A latency increase is not investigated the same way as an incorrect response.

Answering quickly requires telemetry with useful dimensions: version, region, channel, operation type. If metrics only exist as a total, you cannot segment them.

Step 2: follow the trace

With the impact scoped, take a sample of failed or slow operations and follow their path. A distributed trace shows each hop of the request between services, with its duration and result. It shows where the error originates and where the time goes.

When reading traces, look for:

• The first span with an error, not the last. The service that returns the error to the client is often just the messenger.
• Spans with unusual duration compared with healthy traces of the same operation.
• Repeated retries, which amplify load on an already degraded service.
• Calls to external dependencies that time out.

From the trace, jump to the suspect service's logs using the trace ID. That jump is only possible if the logs include the ID. It is an implementation detail that decides whether triage takes minutes or hours.

Step 3: correlate with changes and deployments

When a system that was working stops working, it is common for something to have changed. So, using the start time from step 1, review what changed in that window:

• Application deployments, including those from other teams that share dependencies.
• Configuration changes, firewall rules, certificates or secrets.
• Infrastructure changes: scaling, patches, database or orchestrator version upgrades.
• External changes: provider updates, expirations, traffic shifts from a marketing campaign.

This correlation is fast if change events reach the observability platform and appear as markers on the same charts as the metrics. If they live only in the ITSM tool or the CI system, someone has to look them up by hand. We recommend integrating the change record and the deployment pipeline with observability as standard practice.

Step 4: check saturation

If no change explains the problem, the next suspect is capacity. A saturated resource degrades everything that depends on it, and the symptom can appear far from the resource.

The USE method, proposed by Brendan Gregg, structures this review: for every resource, check utilization, saturation and errors. The resources to review go well beyond CPU and memory:

• Connection pools to databases and external services.
• Worker threads and internal application queues.
• Message queue depth and consumer lag.
• Rate limits on internal and provider APIs.
• Cloud quotas, file descriptor limits, disk space for logs.

An exhausted connection pool, for example, does not show up in server CPU, but it shows up in the latency of every operation that uses it.

Step 5: validate the hypothesis

By this point there is usually a hypothesis. The common mistake is to accept it because it fits the story. A root cause hypothesis is validated with three questions:

1. Does it explain all the observed impact? If the proposed cause is a database node, but operations that do not use it are also failing, the hypothesis is incomplete.
2. Does it explain the start time? The cause must come before or at the same time as the first symptom.
3. Can it be tested? Rolling back the change, isolating the node or freeing the resource should improve the symptom. If you act on the supposed cause and nothing changes, go back to step 2.

Recording each discarded hypothesis and why is also useful. It keeps someone else from rechecking it during the same incident and feeds the post-incident review.

Why the method fails without instrumented code

All five steps depend on data that infrastructure alone does not produce. An APM platform such as Dynatrace or Datadog automatically instruments much of the traffic and the most common runtimes. That is valuable, but it has limits:

• Broken context: if a service hands work to a queue or to its own process without propagating trace context, the trace breaks and step 2 goes blind. OpenTelemetry and the W3C Trace Context standard solve this, but someone has to apply them in the code where automatic instrumentation does not reach.
• Unstructured logs: free text without a trace ID or consistent fields cannot be correlated or filtered quickly.
• Swallowed errors: exceptions caught and discarded, or 200 responses that carry an error, never appear in any metric.
• No business meaning: the platform sees HTTP calls, but does not know that one of them is "authorize payment" or what type of customer made it. Without those attributes, step 1 cannot segment.

An observable platform is not the same as an observable application. Writing observable software is a development decision: structured logs, correlation IDs, context propagation, business metrics and meaningful events. It belongs in each service's acceptance criteria, just like tests.

Triage checklist

Step	Key question	Main signal	Instrumentation requirement
0. Mitigate	Is there a safe action that restores service now?	Recent change history	Ability to roll back deployments
1. Scope the impact	What is failing, for whom, where and since when?	Segmented service metrics	Version, region, channel and operation dimensions
2. Follow the trace	Where does the error or delay originate?	Distributed traces and logs	Context propagation and trace ID in logs
3. Correlate with changes	What changed in the start window?	Deployment and configuration events	Change events sent to the platform
4. Check saturation	Which resource is at its limit?	Utilization, saturation and errors per resource	Metrics for pools, queues and limits
5. Validate the hypothesis	Does it explain all the impact and the start time?	Effect of the corrective action	Hypotheses and actions logged in the incident

After the incident, the blameless review should answer two questions besides the cause: which step of the method took longest, and what data was missing? The answers usually point to a specific instrumentation improvement, which is the most direct way to make the next triage shorter.