Ansible in regulated environments: idempotency, inventories and change control

Automation is also a control

In a regulated organization, every production change has to be explainable: who requested it, who approved it, what exactly was modified and how the result was verified. When changes are made by hand, the evidence depends on screenshots, ticket notes and the operator's memory. It is fragile, and it is hard to reconstruct when the audit arrives.

Ansible changes that picture if it is used with discipline. A versioned playbook describes the expected state, the tool reports which tasks changed something and which did not, and the run leaves a reproducible record. The code becomes part of the control, not just a faster way to get the work done. That benefit does not appear on its own, though: inventories, variables, secrets and runs have to be designed with the auditor and the risk team in mind from the start.

This article covers the decisions that matter most in that design. It does not replace your internal control framework or a reading of the regulation that applies to you (for example, the DORA Regulation in the European Union or your local supervisor's rules), but it does translate their usual expectations on change management into concrete practices.

Idempotency and check mode as evidence

An idempotent module brings a system to a desired state and does nothing if that state already exists. Running the same playbook twice against the same server should produce changes the first time and none the second. That property has an often overlooked benefit: it turns the result of a run into evidence. The final summary (PLAY RECAP) shows, per host, how many tasks ended as ok, how many as changed and how many failed.

Two ansible-playbook options are especially useful in a change process:

• --check runs the playbook in check mode: modules that support it report what they would change without modifying anything.
• --diff shows content differences in files and templates, before and after.

Combined, they produce a preview of the change that can be attached to the record before the maintenance window. The change board or approver no longer evaluates a prose description, but the actual diff that will be applied.

There are nuances worth knowing. Not every module supports check mode, and tasks using ansible.builtin.command or ansible.builtin.shell cannot know whether their command changes anything: they are skipped under --check and, when they run, they report changed. For reliable evidence, use purpose-built modules wherever they exist, declare changed_when on read-only commands and mark with check_mode: false the queries that must also run during the preview. A playbook that reports changes on every run loses its value as evidence: nobody can tell the real change from the noise.

That is also a good health indicator for the repository: after applying a change, a second run in check mode should finish with zero tasks in changed. If it does not, there is configuration drift or poorly written tasks, and both deserve attention.

Inventories and variables per environment

The second pillar is a clear separation between what runs and where it runs. The playbook should be the same in development, testing and production; what changes is the inventory and its variables. A common layout is one inventory directory per environment:

inventories/
  dev/
    hosts.yml
    group_vars/
      all.yml
  prod/
    hosts.yml
    group_vars/
      all/
        vars.yml      # non-sensitive values
        vault.yml     # encrypted with ansible-vault
      web.yml
    host_vars/
      web01.yml

This separation has control implications. First, the target environment is explicit in the command (-i inventories/prod) and can be validated in the pipeline. Second, repository or execution platform permissions can restrict who modifies the production inventory. Third, the diff of a change request shows whether logic (roles and playbooks) or data (an environment's variables) was touched, and each case may call for different reviewers.

A few rules that prevent common problems:

• Define defaults in roles (defaults/main.yml) and override in group_vars only what genuinely differs per environment.
• Avoid defining the same variable at many levels. Ansible has an extensive variable precedence order, and a value arriving from an unexpected place is a classic source of incidents.
• Limit --extra-vars in production to run data (for example, the change identifier), not configuration. Extra vars have the highest precedence and are not versioned.
• If the inventory is dynamic (cloud, CMDB, hypervisor), version the plugin configuration and document which filters determine the production hosts.

Secrets: out of the repository, always

No password, private key or token should be stored in clear text in the repository, not even a private one. Git history keeps everything, and a secret exposed once must be treated as compromised and rotated. There are two reasonable approaches, and many organizations combine them.

Ansible Vault

Ansible Vault encrypts whole files or individual values with AES256. The usual pattern is to keep secrets in an encrypted vault.yml with prefixed names and reference them from a readable vars.yml. That way anyone can search for where a variable is used without decrypting anything:

# inventories/prod/group_vars/all/vars.yml
db_user: app_payments
db_password: "{{ vault_db_password }}"   # the real value lives in vault.yml, encrypted

Vault solves encryption at rest but leaves open the management of the decryption key. Use separate vault identities per environment (the --vault-id option) so that people operating development cannot read production secrets, and keep those keys outside the repository.

External secrets manager

If the organization already has a secrets manager (HashiCorp Vault, CyberArk or the secrets services of AWS, Azure or Google Cloud), the cleanest option is for Ansible to read secrets at run time through lookup plugins from the corresponding collections. Rotation, expiry and access auditing then stay on a platform designed for them, and the repository holds no secrets at all, not even encrypted ones. Execution platforms such as AWX or Red Hat Ansible Automation Platform also let you attach credentials to job templates without exposing them to whoever launches the job.

Either way, set no_log: true on tasks that handle secrets. Without it, a sensitive value can show up in the run output and, with it, in the records later kept for audit.

Every run tied to a change record

A basic change enablement practice is that nothing reaches production without an approved record. With Ansible this can be enforced in the code itself: the playbook requires a change identifier and stops if it is missing or does not match the expected format.

---
- name: Harden the SSH configuration
  hosts: web
  become: true
  vars:
    change_id: ""   # passed with -e change_id=... from the pipeline or platform
  pre_tasks:
    - name: Require a change record with a valid format
      ansible.builtin.assert:
        that:
          - change_id is match('^CHG[0-9]{7}$')
        fail_msg: "Run blocked: a valid change record is missing"
  tasks:
    - name: Disable root login over SSH
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: /usr/sbin/sshd -t -f %s   # the file is not written if the syntax is invalid
      notify: Restart sshd
  handlers:
    - name: Restart sshd
      ansible.builtin.service:
        name: sshd
        state: restarted

The approver's preview and the real run use the same command, with and without --check:

ansible-playbook -i inventories/prod playbooks/ssh.yml --check --diff -e change_id=CHG0012345
ansible-playbook -i inventories/prod playbooks/ssh.yml --diff -e change_id=CHG0012345

The identifier format is an example: adjust it to your service management tool. The control can go further. Instead of only validating the format, a task can query the ITSM tool's API to confirm that the change is approved and that the run happens inside its window. If that query fails, the playbook must stop, not continue.

When runs happen from a central platform rather than an operator's laptop, the link becomes stronger: the job template fixes the playbook, inventory and credentials, and access control defines who can launch it. The change identifier is kept in the job parameters and in its output.

Logging and traceability for audit

Useful evidence answers four questions: what code ran (repository and commit), against which hosts, with what result per task, and who launched it under which change. Ansible covers part of this on its own; the rest depends on how you organize execution.

• Local log. The log_path option in ansible.cfg writes each run's output to a file. It is a useful minimum for runs from a control node, but without protection against modification it does not work as evidence.
• Callback plugins. They let you emit results in other formats or send them to other systems, for example as structured output that a SIEM or log platform can index.
• Execution platform. AWX and Ansible Automation Platform keep job history with user, template, inventory, project revision and full output, and can forward events to an external log aggregator.

# ansible.cfg (excerpt)
[defaults]
log_path = /var/log/ansible/ansible.log
callbacks_enabled = ansible.posix.profile_tasks   # adds per-task duration to the output

For the log to have audit value, ship it to a central repository with defined retention and read-only access for operators. Also correlate the change identifier across every system: ticket, commit, Ansible job and log. With that thread, reconstructing a change stops being an investigation and becomes a query.

Getting-started checklist

1. Every production playbook lives in a repository with mandatory review before merging.
2. One inventory per environment, with stricter permissions for production.
3. No clear-text secrets: Vault with per-environment keys or an external manager.
4. no_log: true on every task that handles sensitive data.
5. A --check --diff preview attached to the change record.
6. The playbook requires the change identifier and records it in its output.
7. Runs from a central platform, not from personal workstations.
8. Centralized logs with defined retention that operators cannot edit.
9. A second check mode run after the change to confirm zero pending changes.

You do not need to implement everything at once. If you are deciding which procedures to automate first, the Value × Complexity framework helps order the backlog: frequent, low-risk changes with a high contribution to audit usually land in Quick wins. As the repository grows, the next step is to treat it as software, with roles, tests and pipelines, the subject of Ansible at scale.