Incident triage with agents: design, controls and traceability

Triage is the moment an alert becomes an incident with an owner, a priority and a first hypothesis. It is work that demands experience and, at the same time, a lot of repetitive reading: opening dashboards, looking up the component in the CMDB, checking for a recent change, seeing whether the same symptom showed up last week. That combination makes it one of the first reasonable candidates for an AI agent, as long as the design puts controls ahead of autonomy.

Why triage is a good first use case

Three characteristics make triage a suitable place to start:

• High volume and repeatable patterns. A large share of alerts correspond to known symptoms with a clear resolver group.
• Verifiable results. An analyst can check in seconds whether the proposed classification and routing make sense, and closing the incident later confirms whether the probable cause was right.
• Containable cost of error. If the agent only suggests, a mistake costs a reassignment, not an outage.

What you should avoid is jumping straight to an agent that reassigns, escalates and takes actions on its own. The value of assisted triage lies in the speed and consistency of the first read, not in removing the analyst.

Reference flow

A design that works well splits the work into stages, each with clear inputs and outputs. That way you can evaluate and correct each stage separately.

1. Alert. The event arrives from the monitoring platform (Dynatrace, Datadog, Prometheus or another) or from the service desk. The agent receives the original payload unmodified.
2. Enrichment. Using read only tools, the agent queries the affected service's metrics, logs and traces for the relevant window; the CMDB record (dependencies, owner, criticality, service hours); changes applied in the last few hours; and recent incidents with similar symptoms.
3. Classification. It proposes category, affected service, impact and urgency according to the priority matrix your organization already uses. It does not invent a new matrix.
4. Probable cause. It states one or more hypotheses, each with the specific evidence supporting it (the metric, the log line, the change) and a confidence level.
5. Routing. It suggests the resolver group based on the CMDB owner and the current assignment rules.
6. Suggested action. If an applicable runbook exists, it references it and proposes the first step. If none exists, it says so explicitly.

The output of the whole flow is a structured note on the ticket, visible to the analyst, who decides whether to accept it, correct it or discard it.

Read only first

The most important design rule is to start with read only permissions on everything: telemetry, CMDB, ticket history and the change repository. The only initial write permission should be adding an internal note to the ticket.

This has practical consequences:

• Use a dedicated service account for the agent, separate from human accounts, scoped to the systems in the flow.
• Give the agent specific tools ("query metrics for service X in this window") instead of generic access to an API or a console.
• Validate each tool's parameters: maximum time windows, allowed services, result limits.
• Treat the content of alerts, logs and tickets as untrusted data. Text inside a log can try to give the agent instructions (prompt injection); the agent should not hold permissions that turn that attempt into damage.
• Review which sensitive data can appear in the logs the agent reads and where it is processed. In regulated industries this must go through security and compliance before deployment.

When and how to expand those permissions is a decision best made with an explicit framework. It is best defined with an explicit scale of autonomy levels, agreed with risk and compliance.

Decision log for audit

If an agent takes part in triage, every one of its interventions must be reconstructable later: what it saw, what it concluded, what it proposed and what the person did with that proposal. This log serves three purposes: audit, improving the agent and team learning.

A minimal record might look like this. It is an illustrative example with fictitious values:

{
  "incident": "INC-000123",
  "agent": "triage-v1.4",
  "model": "model-name-and-version",
  "timestamp": "2025-01-08T03:12:44Z",
  "queries": [
    {"tool": "metrics", "service": "payments-api", "window": "60m"},
    {"tool": "cmdb", "ci": "payments-api"},
    {"tool": "changes", "window": "24h"}
  ],
  "classification": {"category": "performance", "impact": 2, "urgency": 2},
  "hypotheses": [
    {
      "cause": "Database connection pool saturation",
      "evidence": ["metric db.pool.in_use at 100% since 03:01",
                   "change CHG-000456 applied at 02:55"],
      "confidence": "medium"
    }
  ],
  "suggested_group": "Payments platform",
  "runbook": "RB-PAYMENTS-07",
  "human_decision": {"analyst": "analyst-id", "outcome": "accepted_with_changes"}
}

Some recommendations for this log:

• Store the version of the agent, its instructions and the model. Without that you cannot explain why behavior changed.
• Record the human decision as its own field. It is the basis for evaluation.
• Keep the log outside the agent's write scope, with the same retention your policy requires for incident records.
• Require every hypothesis to cite evidence queried in that same run. A hypothesis without evidence should not be shown to the analyst.

How to evaluate accuracy against human triage

Before an analyst trusts the agent's suggestions, you need your own evidence, measured on your data. A simple method is shadow mode: for a defined period, the agent processes the same incidents as the team, but its results are not shown. At the end you compare.

Useful metrics for that comparison:

• Classification agreement: share of incidents where the agent's category and priority match those set by the analyst.
• Routing accuracy: share where the suggested group is the one that actually resolved the incident. This is more informative than comparing with the first human assignment, which is also sometimes wrong.
• Probable cause accuracy: assessed at closure by comparing hypotheses with the documented cause. It helps to distinguish "correct", "partially correct" and "wrong".
• Evidence quality: in a sample reviewed by an experienced analyst, how many cited pieces of evidence were real and relevant.
• Abstentions: how often the agent said "I do not have enough information". An agent that never abstains is a warning sign.

Break results down by incident type. Agents commonly perform well on frequent categories and poorly on rare ones, and a global average hides that difference. Repeat the evaluation whenever you change the model, the instructions or the tools.

Also define, before you start, which result justifies moving from shadow mode to suggestion mode. There is no universal number: it depends on the cost of a wrong assignment in your operation and on how much time the team spends on triage today. What matters is that the criterion is written down and approved by the service owner before seeing the results, so it is not adjusted after the fact. Once in suggestion mode, keep measuring: the rate of suggestions accepted unchanged, the rate of suggestions discarded and, above all, how many accepted suggestions turned out wrong at closure.

Share the results with the analyst team. They are the ones who spot error patterns that an aggregate metric does not show, and their trust in the tool depends on seeing that errors are acknowledged and corrected.

Failure modes and mitigations

The most delicate failure mode is the invented cause: the agent builds a coherent explanation that the data does not support. It is especially dangerous because an analyst in a hurry may accept it. The table summarizes this and other common failure modes in this kind of design.

Failure mode	How it shows up	Mitigation
Invented cause	Plausible hypothesis with no support in telemetry	Require cited, verifiable evidence for each hypothesis; hide hypotheses without evidence
Correlation taken as cause	Blames the latest change purely because of timing	Ask it to state what evidence would rule the hypothesis out; present several hypotheses
Stale context	Routes to a team that no longer owns the service	Show the date of the CI's last update; fix the CMDB as part of closure
Overconfidence	High confidence on new or rare incidents	Allow and measure abstentions; lower confidence when there are no similar incidents
Prompt injection	Text in logs or tickets that changes the agent's behavior	Read only, narrow tools, separate data from instructions
Automation bias	The analyst accepts without reviewing	Show evidence next to the suggestion; periodically review acceptances that proved wrong

None of these mitigations eliminates the risk; they make it visible and measurable. That is the reasonable goal for a first deployment.