Automated remediation: from event to playbook with Event-Driven Ansible

What automated remediation solves, and what it does not

Many of the incidents in an operations center repeat themselves: a service that stops responding, a disk that fills up with logs, a process that consumes memory until the server degrades. The response repeats too: someone receives the alert, opens a session, runs the same steps and documents them in the ticket. If that procedure is already automated in a playbook, the logical next step is for the alert itself to trigger it.

That is automated remediation: detecting a known event and running a known response without waiting for a person to launch it. Applied well, it reduces restoration time for repetitive incidents and frees the team for the problems that genuinely require analysis.

It is worth being precise about its limits. Automated remediation treats known symptoms; it does not replace problem management, which looks for the root cause. If a service needs a restart every week, automating the restart relieves operations, but the problem is still there. A good design records every remediation so that these recurrences are visible and someone investigates them.

How Event-Driven Ansible works

Event-Driven Ansible (EDA) adds the ability to react to events. Its core piece is the rulebook, a YAML file run by ansible-rulebook that combines three elements:

• Event sources: plugins that receive or poll events. There are sources for generic webhooks, Prometheus Alertmanager, Kafka and others, distributed in collections such as ansible.eda. Many monitoring tools can send their alerts via webhook.
• Rules: each rule has a condition evaluated against the event data.
• Actions: what happens when the condition is met. For example, running a playbook, launching a job template or workflow in Ansible Automation Platform, or simply printing the event for debugging.

In Ansible Automation Platform, the EDA controller manages rulebooks as "activations", with their own access control and history. For regulated environments, the recommended action is to launch job templates on the automation controller instead of running local playbooks: each remediation is then recorded as just another job, with its output, its inventory and its managed credentials.

An example rulebook

The following rulebook receives alerts via webhook and defines two responses: a controlled restart for services on an explicit list, and a workflow with human approval for a higher-risk case. Alert, service and template names are examples; the event format depends on what your monitoring tool sends.

---
- name: Application service remediation
  hosts: all
  sources:
    - ansible.eda.webhook:             # receives monitoring alerts over HTTP
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Controlled restart of an allowed service
      condition: >-
        event.payload.alert == "service_down" and
        event.payload.service in ["app-payments", "app-portal"]
      throttle:
        once_within: 30 minutes        # at most one action per host every 30 minutes
        group_by_attributes:
          - event.payload.host
      action:
        run_job_template:
          name: Controlled service restart
          organization: Operations
          job_args:
            limit: "{{ event.payload.host }}"   # only the affected host
            extra_vars:
              service_name: "{{ event.payload.service }}"
              incident: "{{ event.payload.incident_id }}"

    - name: Full disk on a database with prior approval
      condition: event.payload.alert == "disk_full" and event.payload.role == "database"
      action:
        run_workflow_template:         # the workflow includes an approval node
          name: Free up database disk space
          organization: Operations
          job_args:
            extra_vars:
              affected_host: "{{ event.payload.host }}"

For the template to accept limit and extra_vars from the rulebook, it must be configured to prompt for them on launch. And the restart template can be exactly a playbook with pre-checks and post-checks, derived from the runbook the team already used: automated remediation reuses procedures that are already tested, it does not invent new ones.

Two design recommendations. First, normalize events before they reach the rulebook. If every monitoring tool sends a different format, conditions become long and fragile. It is better to agree on a minimal format (alert type, host, service, severity, incident identifier) and adapt it at the source or in an intermediate layer. Second, keep decision logic in the rulebook and execution logic in the playbook. The rulebook decides whether to act and with what data; the playbook decides how, and validates its own conditions again before touching anything. That double validation prevents a malformed or unexpected event from producing a change.

Test conditions before enabling real actions. A rule whose action only prints the event (the debug action) lets you check, with real alerts, which events meet each condition and which do not.

Guardrails: what separates a remediation from a new incident

A playbook that runs on its own, in response to events you do not control, can cause damage very quickly. A misconfigured alert or an event storm can trigger the same restart dozens of times. That is why guardrails are not optional:

• Explicit list of actions. Only approved responses are automated, on named services and hosts. In the example, the condition lists the allowed services; anything else is excluded.
• Rate limit. The throttle option keeps the same rule from acting repeatedly on the same host within a time window. If the problem returns inside that window, insisting is a bad idea: it is a sign that a person is needed.
• Bounded blast radius. The action affects only the event's host (limit), and the playbook works one node at a time. No rule should be able to act on "all servers".
• Human approval for risky actions. Actions that can cause data loss or unavailability (deleting files on a database, failing over a cluster) go through a workflow approval node.
• Kill switch. It must be possible to disable the rulebook activation immediately, and the on-call team must know how.
• Windows and exceptions. During a planned change, expected alerts should not trigger remediations. Coordinate with the maintenance windows in your monitoring tool.

A practical way to decide the level of autonomy is to classify each action by risk:

Level	Example action	Treatment
Diagnostics	Collect logs, metrics and service state	Automatic, no approval
Low risk	Restart one node of a redundant service	Automatic, with rate limit and explicit list
Medium risk	Clean temporary files on an application server	Automatic after a period in observation mode
High risk	Free up space on a database, fail over a cluster	Human approval before running

Even the diagnostics level adds value: when the on-call engineer opens the ticket, the data they need is already attached.

Closing the loop with the ITSM ticket

A remediation that leaves no trace in the service management tool is invisible to incident and problem management. The full loop has four moments:

1. Logging. The incident is opened (by the monitoring tool or by the playbook itself) before acting, and its identifier travels as a variable all the way to the run.
2. Action. The playbook runs the remediation and its checks.
3. Documentation. The playbook adds to the incident what it did, on which host, with what result and the link to the Ansible job.
4. Closure or escalation. If the post-check passes, the incident is resolved with a closure code that identifies automated remediation. If it fails, it is reassigned to the on-call group with the diagnostics attached.

The integration can use dedicated collections (for example, servicenow.itsm for ServiceNow) or calls to the tool's REST API through ansible.builtin.uri. What matters is that updating the ticket is part of the playbook and not a manual step afterwards.

With closures identified, problem management has objective data: which remediations repeat, on which services and how often. That information should feed root cause analysis, not just sit in a report.

Getting started without unnecessary risk

Automated remediation is built gradually. A reasonable path:

1. Choose cases with clear criteria. Apply the Value × Complexity framework to repetitive incidents. Those with high value and low complexity, with a procedure already automated and tested, are the natural Quick wins.
2. Start in observation mode. For a period, rules only log or notify what they would have done. That way you confirm the conditions are right without touching production.
3. Automate diagnostics first. Collecting data is useful and carries almost no risk.
4. Enable low-risk actions with every guardrail active, and review each run during the first weeks.
5. Measure. Remediations run, post-check success rate, escalations and recurrences per service.

Also treat rulebooks like the rest of your automation code: in a repository, with review, testing and promotion between environments, as described in Ansible at scale. A new rule in production is a change, and as such it must go through your change enablement practice.