Monitoring agents: from fixed thresholds to context-aware detection

Almost every command center knows the same scene: dozens of open alerts, half of them duplicates, several with no possible action, and somewhere in the middle an important signal nobody saw in time. The problem is rarely a lack of data. It is how teams decide what deserves attention and how much manual work it takes to understand each alert. This article walks the path from fixed thresholds to context-aware detection, and explains the reasonable role an AI agent can play along the way.

The limits of fixed thresholds

A fixed threshold ("alert if CPU exceeds 85% for 5 minutes") is simple, understandable and easy to audit. That is why it remains useful for hard limits, such as a disk about to fill up or a certificate about to expire. Its problems show up when it is used for everything:

• It ignores timing. 85% CPU can be normal during month-end close and anomalous at dawn on a Sunday.
• It measures causes, not symptoms. High CPU does not always affect users. Latency that doubles without any resource crossing its threshold does.
• It ages badly. Thresholds are set once and the system changes afterwards: more traffic, a new release, different infrastructure. Nobody revisits them.
• It creates cascading noise. A single database failure triggers alerts in every service that depends on it, each one separately.

Google's SRE book stresses a principle worth applying before any technology: alert on symptoms that affect users and require human action, and treat everything else as diagnostic information. An agent does not fix an alerting design that ignores that principle; it only makes it faster.

Baselines and seasonality

The first step beyond fixed thresholds is to compare current behavior with the behavior expected for that moment. That requires a baseline: a model, even a simple one, of how each metric normally behaves.

Most services have visible seasonality: daily cycles (day versus night traffic), weekly cycles (business days versus weekends) and, in banking, insurance or healthcare, business cycles such as month-end close, payroll days or renewal periods. A good baseline accounts for them.

You do not need to start with complex models. Comparing with the same moment the previous week already resolves many false positives. For example, this Prometheus rule alerts when a service's traffic drops below half of what was observed a week earlier:

groups:
  - name: baseline-example
    rules:
      - alert: TrafficLowVsPreviousWeek
        # Compares the current request rate with the same window 7 days ago
        expr: |
          sum(rate(http_requests_total{job="payments-api"}[10m]))
            < 0.5 * sum(rate(http_requests_total{job="payments-api"}[10m] offset 1w))
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Traffic below 50% of the same time last week"

This rule has obvious limits: it fails on holidays and when the previous week also had a problem. Commercial observability platforms (Dynatrace, Datadog and others) offer anomaly detection with automatically learned baselines, and it makes sense to use them. What matters is understanding what each detection compares and documenting it, so you can explain an alert when someone questions it.

Correlating logs, metrics and traces

An isolated anomaly says little. The useful question is which other signals changed at the same time and in which related components. For that, the three signals complement each other:

• Metrics: show that something changed and by how much.
• Traces: show the path a request takes and in which segment time is lost or the error appears.
• Logs: explain what happened at a specific point, with detail the other signals lack.

Correlating them requires shared identifiers: service name, environment, deployed version and, above all, the trace ID in every log. An APM platform captures a lot automatically, but finding the root cause quickly also requires software written to be observable: structured logs, correlation IDs, trace context propagation, business metrics and meaningful events. An observable platform is not the same as an observable application. OpenTelemetry defines semantic conventions for those attributes and is the open reference standard for instrumentation.

This is where an agent starts to add value. Faced with an anomaly, it can query related signals through the service topology, check which dependencies show changes in the same window and separate the original symptom from its cascading effects. It is the same work an experienced analyst does, without having to open ten dashboards.

Adding change and business context

Two kinds of context completely change how an anomaly reads.

Change context

A significant share of incidents follow a change: a deployment, a configuration change, an infrastructure update. If deployment events and approved changes reach the observability platform (or the agent can query them in the change management tool), the first question, "what changed", answers itself. Watch for the opposite bias: the fact that something changed just before does not prove it is the cause.

Business context

A payments service with high latency at three in the morning does not have the same impact as at noon on payroll day. The agent can bring in information such as the business calendar, service hours from the CMDB, maintenance windows and business metrics (approved transactions, completed requests) to estimate impact, not just technical deviation.

Maintenance windows deserve a separate mention: an anomaly during planned maintenance is expected and should not compete for the command center's attention.

How the agent summarizes an anomaly for the command center

The agent's output is not another alert, but a summary that supports a quick decision. A useful, consistent format includes:

1. What is observed: the anomalous signal, the service, since when and how far it deviates from its baseline.
2. Estimated impact: users, transactions or business processes affected, with the source of that data.
3. Related signals: other correlated anomalies, indicating which looks like the origin and which are effects.
4. Context: recent changes, maintenance windows, ongoing business events.
5. Hypotheses: one or more possible causes, each with its evidence and confidence level.
6. Suggested next step: the applicable runbook or the query worth running next.
7. What it does not know: missing data or signals it could not query.

The last item matters as much as the others. A summary that acknowledges its gaps is more useful than one that fills them with assumptions. Every statement in the summary should be traceable to the query that supports it, and the whole process should be logged. In our article on incident triage with agents we describe how to design that log.

In a first stage, the agent should operate with read only permissions and hand its summaries to the operator, who decides. Expanding that scope is a separate decision, best made case by case based on the risk and reversibility of each action.

Keeping false positives in check

Context-aware detection does not mean more alerts. If the command center receives more noise than before, the project has failed even if the technology works. Some practices that help:

• Group before notifying. Correlated anomalies in the same window and topology should arrive as a single event.
• Require persistence. A deviation lasting a few seconds rarely needs human action; set a minimum duration according to the service's criticality.
• Separate notification from recording. Not every anomaly should wake someone up. Many can be recorded for daily review.
• Measure alert quality. For each alert, record whether it led to an action. Rules that almost never do are reviewed or removed.
• Feed back to the agent. When the operator dismisses a summary, the reason ("maintenance", "expected behavior", "duplicate") is valuable input for tuning rules and context.
• Review periodically. Baselines, rules and business calendars change. Schedule reviews, for example quarterly.

The end goal is for every notification reaching the command center to be understandable, actionable and justified. A well designed agent helps you get closer to that goal. Judgment about what to do with each alert stays with the team.