AIOps without the hype: the data it needs and the first year

AIOps entered operations conversations with a broad promise: platforms that learn on their own, detect problems before they happen and cut incidents to a fraction. The reality is more modest and, when framed well, more useful. What these tools reliably do today is apply statistical analysis and machine learning to operational data in order to correlate events, reduce alert noise, detect anomalous behavior in metrics and help narrow down the probable cause of an incident.

None of those capabilities works without quality data, and almost none produces visible results in the first few weeks. This article sets out what to prepare before buying or switching on a platform, what to expect quarter by quarter during the first year and what you should not promise.

What AIOps does and does not do

It helps to separate four capabilities, because they mature at different speeds and depend on different data:

• Noise reduction. Deduplicating repeated alerts, suppressing those that stem from the same symptom and grouping by service. It is the most mature capability and the first to show results.
• Event correlation. Grouping alerts from different tools that belong to the same incident, using time, topology and shared attributes. It depends directly on reliable topology.
• Anomaly detection. Learning a metric's normal behavior, including its daily and weekly cycles, and flagging deviations. It is useful on metrics with stable patterns and noisy on erratic metrics or those with little history.
• Root cause analysis support. Proposing the most likely component or change behind a group of alerts. It is a suggestion for the resolver team, not a diagnosis.

What AIOps does not do is replace the operations team's judgment, fix a poorly instrumented architecture or make up for incident processes that do not exist. If every team opens tickets its own way and nobody records the cause at closure, the platform has nothing to learn from.

The "Monitoring Distributed Systems" chapter of Google's SRE book stresses something that applies here too: alert on symptoms that affect users and keep rules simple and understandable. An AIOps layer on top of poorly designed alert rules just groups the noise more neatly. If your alerts are not yet based on service level objectives, start there; we cover it in alerts without fatigue.

The data it needs before you start

The quality of the results depends less on the algorithm than on four data sets. We recommend reviewing them before any tool evaluation.

Normalized events and alerts

Each monitoring tool emits events with its own format, severity and naming. Correlating them requires a common model: a severity field with agreed values, an identifier for the affected component that matches the CMDB, a synchronized timestamp and a clear status (open or resolved). Without this, two alerts from the same server with different names are treated as different problems.

Reliable topology and CMDB

Topology-based correlation needs to know what depends on what: which application runs on which server, which service calls which database. If the CMDB holds outdated relationships, the platform will group unrelated alerts and split related ones. Automated sources help, such as the monitoring platform's own discovery and distributed traces. OpenTelemetry, whose tracing specification reached stable version 1.0 this year, offers an open way to capture the real dependencies between services.

Change records

Many incidents follow a change. For the platform to suggest "this group of alerts started ten minutes after this deployment", changes must be recorded with their actual execution time, affected components and outcome. Changes made outside the process are invisible to the analysis.

Labeled incidents

To assess whether correlation is right, and to tune the models that produce it, you need an incident history with the affected service, the cause recorded at closure and the link to known problems. This is where the ITIL 4 problem management practice provides what no tool ships with: documented causes and known errors.

Data readiness checklist

Review it with the owner of each source before setting the scope:

• All monitoring tools send events to a common point with an agreed schema.
• Severity is expressed on a single, documented scale.
• Each event identifies the component with the same name the CMDB uses.
• System clocks are synchronized.
• Critical services have their dependencies recorded and recently validated by their owner.
• Production changes are recorded with execution time and affected components.
• Closed incidents have service, cause and category filled in, and you know in what proportion.
• You have a baseline: alerts per day, percentage of alerts with no action taken and mean time to detect and to resolve.

If several items fail, the first quarter of the project will be about data, not algorithms. Better to know that before signing.

The first year, quarter by quarter

A realistic plan moves from the simplest to the most demanding. Capabilities that depend on topology and history come after those that only need well-formed events.

First quarter: data and baseline

Integrate the event sources, normalize the schema, clean up CMDB relationships for critical services and measure the starting point. The deliverable is not fewer alerts: it is an inventory of sources, a common schema and a baseline to compare against. It is also the time to choose a narrow scope, for example two or three critical services, rather than the whole operation.

Second quarter: noise reduction

Turn on deduplication, suppression and grouping by service within the chosen scope. This is where the first measurable results appear: fewer alerts reaching a person for each real incident. Rules are tuned weekly with the team that runs the services, since they are the ones who can tell a useful grouping from one that hides information.

Third quarter: assisted correlation

With clean topology and recorded changes, the platform can propose which alerts belong to the same incident and which recent change is the most likely candidate. At this stage we recommend that the suggestion appears in the ticket and that the resolver marks whether it was right. That answer is the data point that lets you measure accuracy. The investigation method is still the team's: form hypotheses and test them, as the "Effective Troubleshooting" chapter of the SRE book puts it. We describe it in root cause in distributed systems.

Fourth quarter: anomalies and tuning

Turn on anomaly detection only for metrics with enough history, stable patterns and a direct link to a symptom users notice. Review the year's results against the baseline, decide which services to onboard next and which automated responses make sense. Those cases go into the backlog under the same criteria as any other automation, as we explain in governing the automation backlog.

What not to expect in the first year

• Incident prediction. Anticipating failures requires long history, consistent data and failures that repeat in recognizable patterns. In the first year, the reasonable goal is to detect earlier, not to predict.
• Automatic root cause. The platform suggests candidates. Confirming the cause is still the job of the team and of problem management.
• Hands-off remediation. Running automated actions based on a correlation whose accuracy has not been measured shifts the risk rather than reducing it. Measure first; automate later, starting with reversible actions.
• Full coverage. A few well-instrumented services teach you more than the whole operation with uneven data.
• Reduction figures promised up front. The effect depends on the starting point. If someone guarantees a percentage without knowing your data, ask them to justify it against your baseline.

How to measure whether it works

Metrics are defined in the first quarter, together with the baseline, and reviewed every month. We suggest a short set:

Metric	What it shows
Alerts reaching a person per incident	Effect of noise reduction
Percentage of alerts with no action taken	Quality of the source rules
Correlation groups confirmed by the resolver	Correlation accuracy
Suggested cause accepted at closure	Usefulness of root cause support
Mean time to detect and to resolve	Impact on the service
Reviewed anomalies that turned out to be real	Signal to noise ratio of the models

Two cautions. First, mean times blend very different incidents; look at them by service and by severity. Second, fewer alerts can be bad news if something that mattered was suppressed. That is why every suppression rule needs an owner and a periodic review.

Where it fits in service management

AIOps is not a new practice but a support layer for practices that already exist. In ITIL 4 it feeds incident management (faster detection and grouping), problem management (recurring patterns and known error candidates) and monitoring and event management. In SRE terms, it helps on-call engineers receive fewer interruptions with no value, one of the goals Google's book sets for any monitoring system.

That perspective shapes the buying decision. The question is not which platform has the most algorithms, but which practice you want to improve, what data you have today and how you will measure the change. If the answers are clear, the first year delivers concrete results. If they are not, the first quarter is the time to clarify them.