From runbook to playbook: automating auditable procedures

The runbook as a starting point

Almost every IT operation has runbooks: documents that describe, step by step, how to restart a service, rotate a certificate or free up disk space on a server. They are valuable because they capture operational knowledge. They also have well-known problems: they go stale, each operator interprets them differently, some steps get skipped under pressure, and the evidence of what was done ends up scattered across chat, the ticket and the console.

Turning a runbook into an Ansible playbook is not only about saving time. It turns an instruction someone has to follow into a procedure that always runs the same way, validates its own conditions and records every step. Done well, the playbook is more auditable than the manual procedure it replaces.

This article uses a concrete and common example: restarting an application service with checks before and after. The method applies to any repeatable procedure.

Choosing the candidate with Value × Complexity

Not every runbook deserves automation, and certainly not all at once. To choose, we apply the Value × Complexity framework, which has three steps: define, classify and prioritize.

Define. Each candidate is documented in a profile: process, trigger, frequency, volume, systems involved, owner, expected outcome and current risk. For our example, the profile might read: restart of the payments service, triggered by a degradation alert or a request from the application team, with a frequency of, say, eight times a month, across four servers, owned by the operations team.

Classify. Both axes are scored from 1 to 5. Value is the average of effort saved, risk and error reduction, impact on the customer or the SLA, and contribution to compliance or audit. Complexity is the average of the number of systems and integrations, process variability, quality of available data and access, production change risk and third-party dependency. An example score:

Axis	Criterion	Score (example)
Value	Effort saved	3
Value	Risk and error reduction	4
Value	Impact on the customer or SLA	4
Value	Contribution to compliance or audit	4
Complexity	Systems and integrations	2
Complexity	Process variability	2
Complexity	Data quality and access	2
Complexity	Production change risk	3
Complexity	Third-party dependency	1

Prioritize. The cutoff is 3: an average of 3 or more counts as high, and below 3 as low. With a Value of 3.75 (high) and a Complexity of 2 (low), the candidate lands in Quick wins: it goes first. Had the score shown high complexity (for example, because the restart requires coordination with an external provider), it would be a Strategic bet and should be split into smaller deliveries.

Breaking down the runbook: from step to task

Before writing any YAML, rewrite the runbook as a list of atomic steps, each with its own success criterion. Ambiguous steps ("check that everything looks fine") are exactly the ones that need precision: what is checked, against which threshold, and what happens if it is not met.

Runbook step	Playbook task	Validation
Check disk space	Query /var usage	Usage below 90 %, or stop
Confirm the service exists	ansible.builtin.service_facts	The service appears in the host facts
Restart one node at a time	serial: 1 and ansible.builtin.service	If a node fails, the next one is not touched
Check that it responds	ansible.builtin.uri with retries	HTTP 200 within the time limit
If it does not respond, escalate	rescue section	Collects the log and fails with an actionable message

This exercise usually surfaces steps the runbook took for granted and nobody had written down. That is a gain in itself, even before automating. Do it with the people who run the procedure today, not only with whoever documented it: the differences between the two versions are usually the exceptions the playbook will need to handle.

The playbook: pre-checks, change and post-checks

The resulting playbook follows the runbook's structure, but every step has an explicit condition. Service name, port and threshold are examples:

---
- name: Controlled restart of the payments service
  hosts: app_payments
  become: true
  serial: 1                  # one node at a time: the rest keep serving
  max_fail_percentage: 0     # if a node fails, the run stops
  vars:
    service_name: app-payments
    health_url: http://localhost:8080/health
  tasks:
    - name: Pre-check. Query /var usage
      ansible.builtin.command: df --output=pcent /var
      register: var_usage
      changed_when: false    # read-only
      check_mode: false      # also runs under --check

    - name: Pre-check. Require /var usage below 90 %
      ansible.builtin.assert:
        that:
          - (var_usage.stdout_lines[1] | trim | replace('%', '') | int) < 90
        fail_msg: "/var usage too high: not restarting"

    - name: Pre-check. Gather service state
      ansible.builtin.service_facts:

    - name: Pre-check. Confirm the service exists
      ansible.builtin.assert:
        that:
          - (service_name ~ '.service') in ansible_facts.services

    - name: Restart with post-check
      block:
        - name: Restart the service
          ansible.builtin.service:
            name: "{{ service_name }}"
            state: restarted

        - name: Post-check. Wait for HTTP 200 from the health endpoint
          ansible.builtin.uri:
            url: "{{ health_url }}"
            status_code: 200
          register: health
          until: health.status == 200
          retries: 10            # up to 10 attempts
          delay: 6               # 6 seconds between attempts
      rescue:
        - name: Collect the last lines of the service log
          ansible.builtin.command: journalctl -u {{ service_name }} -n 50 --no-pager
          register: service_log
          changed_when: false

        - name: Show the log so it can be attached to the ticket
          ansible.builtin.debug:
            var: service_log.stdout_lines

        - name: Stop with an actionable message
          ansible.builtin.fail:
            msg: "Post-check failed on {{ inventory_hostname }}. Escalate to on-call."
      always:
        - name: Record the outcome in the output
          ansible.builtin.debug:
            msg: "{{ inventory_hostname }}: health status {{ health.status | default('no response') }}"

Some decisions deserve explanation. serial: 1 reproduces the runbook rule of never restarting every node at once, and max_fail_percentage: 0 ensures that a failure on the first node protects the others. Read-only queries carry changed_when: false so that the final summary shows only the restart as a change. And the post-check does not stop at confirming the process started: it queries the health endpoint, which is what actually shows whether the service is serving requests.

When something fails: block, rescue and always

In a runbook, the error section is usually the weakest: "if it fails, contact the owner". In a playbook, failure handling is part of the design. The block, rescue and always structure works much like try, catch and finally:

• block groups the change tasks.
• rescue runs only if a task in the block fails. This is where the response to the error goes: collect diagnostics, attempt a rollback if a safe one exists, and make the failure explicit.
• always runs in every case. It is the place to record the outcome or notify.

One important detail: if rescue finishes without errors, Ansible considers the host recovered and the run continues. That is why the example ends the rescue with ansible.builtin.fail. Hiding a failure behind a silent rescue is worse than having no automation, because the evidence says everything went fine.

Also decide what to do with the pending nodes. With serial: 1 and max_fail_percentage: 0, a failure stops the run before the next node is touched. For procedures where a partial failure leaves the system inconsistent, any_errors_fatal: true stops the run on every host.

Where to keep human approval

Automating does not mean removing people from the process. It means deciding precisely where they add control. Some guidelines:

• Before running, approval is still needed when the procedure is a normal change under your change enablement practice. Once the playbook has proven stable, many procedures can be registered as pre-approved standard changes.
• During the run, if a pause is needed for someone to confirm (for example, between the first node and the rest), platforms such as AWX or Ansible Automation Platform provide approval nodes in workflows. The ansible.builtin.pause module can also ask for confirmation, but it only works in interactive runs from the console.
• After a failure, the decision to retry or roll back should stay with a person, using the diagnostics the playbook collected.

Approval must also leave evidence: who approved, when and for which run. A "yes" in a chat does not serve that purpose; an approval node on the platform or a status in the change record does.

From playbook to auditable procedure

The work does not end when the playbook works. To make it an auditable procedure, close these points:

1. The playbook lives in a repository with mandatory review, and the original runbook is updated to state that the procedure is now automated and how to launch it.
2. Runs happen from a central platform, with access control over who can launch them.
3. Each run is linked to a ticket or change record, and its output is kept for the retention period your control framework requires.
4. A manual fallback procedure exists in case the automation platform is unavailable.
5. The playbook is tested in a non-production environment before every modification.

Finally, rescore the backlog. Automating one procedure changes the value and complexity of the next ones: a health-check role already written lowers the complexity of every procedure that reuses it. To see how to bring these same checks into an automated response to alerts, continue with Automated remediation with Event-Driven Ansible.