Problem management: the practice that actually reduces incidents

Many IT operations are very good at firefighting. They restore quickly, coordinate major incidents well and have committed on-call teams. And yet, month after month, incident volume does not go down. The reason is almost always the same: incident management works, but problem management does not exist, or exists only on paper.

An incident is not a problem

The distinction is easy to state and hard to sustain in practice. An incident is an interruption or degradation of a service; the goal is to get the service back to normal as fast as possible, even with a patch. A problem is the cause, actual or potential, of one or more incidents; the goal is to understand it and remove it, or at least reduce its impact.

The two practices run at different speeds. Incident management works in minutes and hours, under pressure. Problem management works in days or weeks, calmly and with data. When they get mixed, both lose: the incident stays open "until we know what happened" and restoration metrics inflate, or the problem is considered solved the moment service is back and nobody investigates.

ITIL 4 organizes the practice into three phases worth keeping in mind: problem identification, problem control (analyzing problems and documenting workarounds and known errors) and error control (managing known errors until they are permanently fixed or formally accepted).

Reactive and proactive: two ways in

Reactive problem management starts from incidents that have already happened. Typical triggers are a major incident, a recurring incident or a cluster of incidents with similar symptoms. This is where almost every organization begins, and rightly so: the pain is visible and the value is easy to explain.

Proactive problem management looks for problems before they cause incidents users can see. Its sources are trend analysis of incident history, alerts that fire and clear on their own, capacity warnings, vulnerabilities published by vendors and known errors reported by software makers.

A practical rule to start: spend most of the capacity on reactive work during the first months, until the known error record has useful content, and then reserve a fixed slot (for example, a trend review every two weeks) for proactive work. If proactive work has no time assigned, it never happens.

When the list of open problems grows, you have to prioritize. A simple criterion combines three factors: how many incidents the problem caused in the period, which services it affected, and how much effort the workaround consumes today. A problem that causes few incidents but forces the team to intervene manually every night may be worth more than a noisy, low-impact one. Reviewing this prioritization with each service owner keeps the technical team from deciding alone what matters.

Known errors and workarounds

A known error is a problem that has been analyzed but does not yet have a permanent fix. A workaround is a way to reduce or avoid the impact in the meantime: restarting a service, rerouting traffic, applying a manual adjustment.

The known error record is one of the highest-return assets in an operation, as long as it meets three conditions:

• The service desk can access it. If only specialists consult it, it does not reduce restoration times.
• It describes symptoms the way users see them. "Account statement does not load" is easier to search than "connection pool timeout in service X".
• It has an owner and a review date. A workaround that becomes permanent without anyone deciding it is hidden technical debt.

Every known error ends in one of two ways: with a change that fixes it, handled through change enablement, or with an explicit decision to accept the risk because fixing it costs more than living with it. Both are valid. Leaving it open indefinitely is not.

Root cause techniques and their limits

There is no universal technique. Each fits a kind of situation, and they all share one risk: stopping at the first plausible explanation. In complex systems there is rarely a single cause; there are usually several contributing factors that line up.

Technique	When it helps	Main limit
Five whys	Contained problems with a fairly linear causal chain	Follows a single path; depends on what the group already knows and can end at "human error"
Ishikawa (fishbone) diagram	Problems with many possible causes; organizes brainstorming by category	Produces hypotheses, not evidence; each branch must be validated with data
Timeline analysis	Major incidents and failures that span several systems or teams	Requires synchronized clocks and complete records; without data, it is rebuilt from memory

In practice, combine them. The timeline establishes what happened and in what order. The Ishikawa diagram opens up the range of possible causes. The five whys dig into the ones the evidence supports.

Two method recommendations. First, keep the analysis blameless: the goal is to understand why the system allowed the error, not who made it. If the conclusion is "the person made a mistake", the next question is what made that mistake possible. Second, make every conclusion cite its evidence: a metric, a log, a recorded change. A root cause without evidence is an opinion.

Observability speeds up the analysis

Much of the time in a problem analysis goes into gathering data: what changed, when the degradation began, which components took part in the failed transaction. This is where observability makes the difference.

Logs, metrics and traces are complementary signals. Metrics show when and how much; traces show where a transaction went and where it slowed down; logs explain what happened at each point. An APM platform, such as Dynatrace, Datadog or others, makes the infrastructure observable and captures a lot automatically, including correlation across services.

But an observable platform is not the same as an observable application. For root cause analysis to be fast, the software has to be written to be observable:

• Structured logs, with consistent fields instead of free text.
• Correlation identifiers that let you follow an operation end to end.
• Trace context propagation across services, queues and third-party calls.
• Business metrics (operations completed, rejected, abandoned), not only technical ones.
• Meaningful events, such as deployments and configuration changes, recorded on the same timeline.

OpenTelemetry is the open reference standard for instrumenting these signals. With that foundation, timeline analysis stops being a reconstruction from memory and becomes a query. That is also why problem findings should feed the development backlog: many hard-to-find root causes reveal, along the way, what was missing from the instrumentation.

Metrics that show whether the practice works

Problem management is judged by its effect on incidents, not by the number of analyses performed. These are the metrics we recommend tracking:

• Recurrence: incidents tied to problems already analyzed. This is the main metric; if it does not drop, the analysis is not reaching the cause.
• Incidents resolved with a known error: share of incidents the service desk resolved by applying a documented workaround.
• Time to known error: days from opening the problem to having a published workaround.
• Age of open known errors: how many have gone longer than the organization considers acceptable without a fix or formal acceptance.
• Share of proactive problems: how many problems were opened before there was an incident visible to users.

Avoid the temptation to measure "problems closed per month". It rewards closing fast, not closing well.

How to start in four steps

1. Define the opening criterion. For example: every major incident, and every incident with the same symptom that occurs three times in a month.
2. Assign a practice owner. This person does not have to analyze everything, but must make sure every problem has an owner, a date and follow-up.
3. Publish the known error record. Even if it starts with few entries, it must be integrated into the service desk tool.
4. Review recurrence every month. A short meeting with incident, change and development teams, focused on the five most repeated symptoms.