How to measure the value of an automation before building it

Measure before, not after

Most teams measure the value of an automation once it is already in production, if they measure it at all. By then the investment decision has been made. What helps is estimating value before building, with a method simple enough to apply to dozens of candidates and explicit enough to debate.

In our Value × Complexity framework, the Value axis is the average of four factors, each scored from 1 to 5:

• Effort saved: hours per month freed.
• Risk and error reduction.
• Customer or SLA impact.
• Compliance or audit contribution.

This article explains how to score each one consistently, so that a 4 means the same thing whether the network team or the database team assigns it.

A rubric for the four factors

A rubric does not remove judgment, but it anchors it. The table describes the extremes and the midpoint; 2 and 4 are intermediate values. The hour thresholds are reference values we propose as a starting point: adjust them to the size of your team and operation.

Factor	1	3	5
Effort saved	Fewer than 5 hours per month freed.	Between 15 and 40 hours per month.	More than 80 hours per month.
Risk and error reduction	The task rarely produces errors, or its errors are minor and reversible.	Occasional errors that cause rework or minor incidents.	One error can cause a service outage, data loss or a security exposure.
Customer or SLA impact	No visible effect for users or customers.	Shortens response times for internal requests.	Directly affects availability or an SLA committed to customers.
Compliance or audit contribution	No link to controls or audits.	Produces useful evidence for an internal control.	Applies or evidences a control required by regulation, or closes an open audit finding.

For effort saved, the intermediate reference values are: 2 between 5 and 15 hours per month, and 4 between 40 and 80 hours per month.

Two practical rules help when scoring with the rubric. First, whoever scores must be able to justify the number in one sentence; if they cannot, the score is an opinion. Second, when torn between two values, pick the lower one and note what data would justify raising it.

How to estimate hours freed

Hours are the easiest factor to quantify and, for that very reason, the easiest to inflate. We recommend always estimating them with the same formula:

Hours per month = frequency × volume × duration

• Frequency: how many times per month the task occurs.
• Volume: how many items are processed each time (servers, accounts, files, requests).
• Duration: how long each item takes, measured rather than remembered.

For example, take a daily backup review. It runs on 22 business days a month, 15 backup jobs are checked each day and each check takes about 4 minutes. That gives 22 × 15 × 4 = 1,320 minutes, or 22 hours per month.

From that gross number, subtract the effort the automation does not remove:

1. Residual effort: someone will still review the exceptions the automation flags. In the example, assume 2 hours per month.
2. Maintenance: rotating credentials, changing versions, adjustments as the process evolves. Assume 1 hour per month.

Net savings come to 19 hours per month, which the reference rubric scores as a 3. The gap between gross and net looks small here, but in fragile automations or those with many exceptions it can eat half the savings.

Three warnings about the estimate:

• Measure duration. Ask whoever performs the task to time it for one or two weeks. Memory tends to underestimate boring tasks and overestimate annoying ones.
• Use typical frequency, not the peak. If the task spikes at month end, reflect that as such rather than treating every week as month end.
• Look at how hours are spread. Twenty hours concentrated in one person free real capacity. Twenty hours spread across twenty people, one hour each, free much less than the number suggests.

How to value risk and compliance

Risk reduction is the most underestimated factor, because it never shows up on a timesheet. A ten-minute monthly task may be the one that, done wrong, takes a service offline. Manual certificate renewal is a classic example: little effort, high risk.

To score risk without inventing figures, we recommend looking at concrete evidence:

• Incident and problem history: are there incidents caused by a mistake in this manual task?
• Near misses: errors caught in time that could have escalated.
• The organization's risk matrix: if one exists, use its probability and impact scale instead of creating a parallel one.
• Single-person dependency: if only one person knows how to run the task, the risk is higher even if it has never failed.

Compliance contribution is valued by asking which specific control the automation supports. It may be an internal control, a requirement of a standard such as ISO/IEC 27001 or a regulatory obligation. For financial organizations operating in the European Union, for example, the DORA Regulation sets ICT risk management obligations that make traceable, consistent evidence especially valuable.

An automation contributes to compliance in three ways: it runs the control the same way every time, it records each run and it reduces reliance on someone remembering to do it. If the automation leaves no auditable record, its compliance contribution is low even when the process is regulated.

Customer or SLA impact

This factor answers a simple question: if the automation works, does anyone outside the team notice? It may be an external customer who no longer suffers an outage, or an internal user whose request is resolved in minutes instead of days.

To score it, start from the service level agreements in force. If the manual task sits on the critical path of a committed SLA, such as service restoration time, impact is high. If it only speeds up an internal process with no formal commitment, it is medium. If nobody outside the team perceives the difference, it is low.

Common mistakes when measuring value

• Counting only hours. The most common mistake. It leaves out risk reduction and compliance, which are often what justify automating small but critical tasks.
• Using gross hours. Without subtracting residual effort and maintenance, value is overstated.
• Confusing hours freed with budget savings. Freeing hours lets you reassign capacity; it does not automatically cut costs. Promising the latter when only the former will happen erodes the program's credibility.
• Counting the same benefit twice. If a regulatory control reduces a risk, decide where it weighs more and do not score it high on both for the same reason.
• Ignoring the time horizon. If the system involved is retired in six months, the accumulated value of automating it is lower, however high the monthly savings.
• Scoring by who is asking. Value is measured on the process, not on the requester's rank.

From factor to score: an example

Illustrative example. Back to the daily backup review, with fictional data:

• Effort saved: 19 net hours per month, score 3.
• Risk reduction: an undetected failed backup can mean data loss during an incident, score 4.
• Customer or SLA impact: the ability to restore services depends on valid backups, score 3.
• Compliance contribution: backup verification is often a control reviewed in audits and the automation leaves a daily record, score 4.

Value is (3 + 4 + 3 + 4) / 4 = 3.5: high value. Had only hours been counted, the candidate would have sat on the boundary. With all four factors it is clear why it deserves attention. The next step is to measure its complexity, covered in the factors that sink an automation.