Ansible at scale: roles, collections, testing and pipelines

When a playbook repository stops scaling

Most Ansible initiatives start the same way: an administrator writes a playbook to handle a repetitive task, it works, and ten more follow. While a single person maintains them, the mess is tolerable. The problem arrives when automation becomes critical and several teams use it. The signs are easy to recognize:

• The same logic (configuring NTP, installing the monitoring agent, hardening SSH) is copied across several playbooks with small differences.
• Nobody knows for sure which version of a collection or of ansible-core each run uses.
• A change meant for one team breaks another team's automation, and it is discovered in production.
• Testing means running against a development server "to see what happens".

The answer is not a new tool, but applying to automation what is already expected of business software: structure, reuse, automated tests, versioning and controlled promotion. In a regulated environment this is also part of change control.

Repository structure

There is no single correct structure, but there are principles: keep playbooks, roles and inventories apart, declare external dependencies and keep the tool's configuration next to the code. A reasonable starting point:

automation/
  ansible.cfg                  # tool configuration, versioned
  .ansible-lint                # repository quality rules
  collections/
    requirements.yml           # external collections with pinned versions
  inventories/
    dev/
    qa/
    prod/
  playbooks/
    site.yml
    monthly_patching.yml
  roles/
    base_linux/
      defaults/main.yml        # default values, can be overridden
      handlers/main.yml
      meta/main.yml            # role metadata and dependencies
      tasks/main.yml
      templates/
      molecule/default/        # role test scenario

Playbooks should be thin: they select hosts and call roles. Logic lives in roles, which receive variables and do not know which environment they run in. That rule is what makes it possible to test a role in isolation and reuse it elsewhere.

External dependencies are declared in requirements.yml and installed with ansible-galaxy collection install -r collections/requirements.yml. Pinning versions prevents a new collection release from changing production behavior without anyone deciding it:

---
collections:
  - name: ansible.posix
    version: ">=1.5.0,<2.0.0"   # bounded range: accepts fixes, not major changes
  - name: community.general
    version: ">=8.0.0,<9.0.0"
  - name: mycompany.platform       # internal collection (example name)
    version: "2.3.1"               # exact version, published to a private repository

Roles and collections: when to use each

A role packages tasks, handlers, templates and variables to handle one specific responsibility. A collection is Ansible's distribution unit: it groups roles, modules, plugins and playbooks under a namespace (namespace.name), with a version declared in its galaxy.yml file. Since Ansible content is distributed in collections, fully qualified names (FQCN), such as ansible.builtin.copy or mycompany.platform.base_linux, are the recommended way to refer to modules and roles.

Aspect	Role in the repository	Collection
Content	Tasks, handlers, templates, variables	Roles, modules, plugins and playbooks
Distribution	Alongside the code that uses it	Packaged artifact (Galaxy, Automation Hub or a private repository)
Versioning	The repository's	Its own, declared in galaxy.yml
Typical tests	ansible-lint and Molecule	ansible-lint, Molecule and ansible-test
Best fit	Logic used by a single team or repository	Logic shared across teams or with custom modules

A natural path is to start with roles inside the repository and extract to an internal collection the ones several teams need. The collection is created with ansible-galaxy collection init, packaged with ansible-galaxy collection build and published to a private repository. From then on, each team consumes an explicit version and decides when to upgrade.

Static quality with ansible-lint

ansible-lint checks playbooks, roles and collections against a set of rules: FQCN usage, unnamed tasks, commands without changed_when, implicit file permissions, YAML syntax and other practices. It groups rules into profiles of increasing strictness (from min to production), which allows gradual adoption in an existing repository.

# .ansible-lint
---
profile: production       # the strictest profile; start with "basic" on an older repository
exclude_paths:
  - .cache/
  - collections/ansible_collections/   # third-party content installed locally
skip_list: []             # every exception is documented and reviewed

Two recommendations. First, run ansible-lint on the developer's machine too (for example, with a pre-commit hook), not only in the pipeline: the cheap error is the one fixed before pushing. Second, treat exceptions as debt: every skipped rule should have a written reason and a review date.

Complement the analysis with ansible-playbook --syntax-check on the main playbooks. It is fast and catches structural errors before any test against real hosts.

Testing roles with Molecule

Static analysis does not tell you whether a role works. That is what Molecule is for: it creates test instances (containers or virtual machines, depending on the driver), applies the role and verifies the result. The molecule test command runs a full sequence that includes, among other steps, creating the instance, applying the role (converge), applying it again to check idempotency, running verification and destroying the instance.

The idempotency step is the most valuable one for operations: if the second application reports changes, the test fails. It catches exactly the kind of task that later pollutes evidence in production. Verification can be written as a playbook with assertions:

# roles/base_linux/molecule/default/verify.yml
---
- name: Verify the base_linux role
  hosts: all
  gather_facts: false
  tasks:
    - name: Gather service state
      ansible.builtin.service_facts:

    - name: Check that the time service is running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['chronyd.service'].state == 'running'
        fail_msg: "chronyd is not running"

There are practical limits. Testing systemd-managed services inside containers requires images prepared for it, and some roles (those touching the kernel, storage or hardware) can only be tested realistically on virtual machines. Start with the most reused roles and accept that coverage grows gradually.

For collections with custom modules or plugins, ansible-test provides sanity, unit and integration tests, and it is the tool the Ansible project itself uses for its content.

Pipeline: stages and promotion between environments

With structure and tests in place, the pipeline ties everything together. Whatever CI server you use, the stages usually look like this:

1. Static analysis. yamllint, ansible-lint and --syntax-check on every change request. Merging is blocked if it fails.
2. Role tests. molecule test on the modified roles.
3. Packaging. A versioned artifact is built: the collection, an execution environment, or both.
4. Publishing. The artifact is published to a private collection repository or a container registry.
5. Deployment to development and QA. First with --check --diff, then the real run, followed by smoke tests.
6. Promotion to production. The same version, with approval and a change record.

The key principle is to promote artifacts, not branches. What reaches production must be exactly what was tested in QA: the same collection version and the same execution environment. If each environment rebuilds from a branch, nothing guarantees an identical result.

Execution environments help here. They are container images that include ansible-core, the collections and their Python dependencies. They are built with ansible-builder from a definition file and run with ansible-navigator or from Ansible Automation Platform. The engine version then no longer depends on whatever is installed on each control node.

Plan for rollback too. If a new version fails in production, the fastest response is often to rerun the previous version of the artifact, which is still published and already tested. That only works if roles are idempotent, if the new version did not introduce changes the previous one cannot revert, and if the pipeline lets you choose which version to deploy instead of always taking the latest. Document that procedure and rehearse it in QA before you need it: a rollback that was never tested is a hypothesis, not a plan.

Versioning and content governance

Collections use semantic versioning, and it pays to follow it strictly because consumers depend on it:

• Major: incompatible changes, such as renaming a role variable or removing a parameter.
• Minor: new compatible functionality, such as an additional option with a default value.
• Patch: fixes with no interface changes.

Ship every version with a readable changelog and treat role variables as a public interface: documented in a README or in the role's argument specifications (meta/argument_specs.yml), which also validate types at run time. Assign owners per collection and mandatory reviewers for changes to shared content.

Finally, measure. A few simple indicators give visibility into the program: share of roles with Molecule tests, time from merge to production, failed runs per version and number of active ansible-lint exceptions. The point is not to chase a number, but to see where debt is building up. And if you need to decide what to automate next, the Value × Complexity framework orders the backlog with explicit criteria.