AI-ready data before the first pilot

Many AI pilots get approved on the strength of a convincing demo and stall a few weeks later over a question nobody asked at the start: are we allowed to use this data? Sometimes the problem is technical (the key field is empty in a third of the records). Other times it is about governance: nobody knows who owns the table, which system it comes from or whether it holds personal data that should never leave its perimeter.

In a regulated industry those questions cannot wait. A model that learns from wrong data gives wrong answers with great confidence, and a pilot that uses data without authorization is a compliance incident even if it works well. This article proposes four foundations to put in place before the first pilot: catalog, quality, lineage and permissions. You do not need them across the whole organization; you need them for the data the pilot will use.

Why pilots stumble over data

The pattern is familiar. The team picks an attractive use case, gets a data extract "just to test", trains or configures the model and gets promising results. When it is time to move to production, Risk, Security and Compliance start asking: where did that extract come from, who authorized it, does it include customer data, can it be repeated every day at the same quality? If the answers do not exist, the pilot starts over or gets shelved.

The cost is not just lost time. A manual extract sitting in a shared folder is a risk in itself: it copies data outside its controls, has no owner and nobody deletes it. That is why it pays to reverse the order: prepare the data for the use case first, then build the pilot. The preparation does not have to be a multi-year program. For a well-scoped pilot, it can take a few weeks if the questions are asked from day one.

A good starting point is the NIST AI Risk Management Framework. Its Map function asks you to understand the context of the AI system, including the data it uses and its limitations; its Measure function asks you to assess with metrics and document the results. The four foundations in this article are a concrete way to answer both.

Catalog: know what exists and who owns it

The catalog answers two basic questions: what data exists for the use case, and who is accountable for each dataset. Without an owner there is nobody to ask for permission, nobody to report a quality issue to and nobody to approve a schema change.

You do not need an enterprise catalog tool to get started. For the pilot, each dataset should have a record with the essentials:

• Name and description in business language, not just the technical table name.
• Business owner (decides how it is used) and technical custodian (operates it).
• Source system and refresh frequency.
• Classification under the organization's policy (for example, public, internal, confidential, restricted) and whether it contains personal data.
• Definitions of key fields: what "active customer" or "transaction date" means in that dataset.

The last point looks minor and is not. Two departments can use the same term with different definitions, and a model trained on one definition gets evaluated against the other. That record is also the seed of the enterprise catalog: every pilot that fills it in leaves the ground better prepared for the next one.

Quality: dimensions you can measure

"The data is good" is not a criterion. Quality is expressed in measurable dimensions, with a threshold agreed with the owner before you start. The ISO/IEC 5259 series (Artificial intelligence. Data quality for analytics and machine learning) offers a recognized framework: it defines a data quality model and measures for analytics and machine learning, and extends the classic ISO/IEC 25012 model with characteristics specific to these uses, such as representativeness, diversity and balance.

For a first pilot we recommend choosing a few dimensions and measuring them well. The table below is illustrative; actual thresholds depend on the case.

Dimension	Question it answers	Example measure (illustrative)
Completeness	Are key fields missing values?	Percentage of records with the field populated
Accuracy	Does the data reflect reality?	Match rate against a hand-validated sample
Consistency	Do two systems say the same thing?	Differences in a reconciliation across sources
Currentness	Does it arrive in time for the use?	Hours between the event and its availability
Representativeness	Does it cover the real population of the case?	Distribution by segment compared with production

Representativeness deserves special attention in AI. A dataset can be complete and accurate, but if it only covers one type of customer, one channel or a period with no seasonality, the model will learn a world that is not the real one. In banking, insurance or healthcare this also has a fair-treatment dimension that Risk and Compliance will want to review.

Measuring once is not enough. If the pilot moves forward, quality must be watched continuously, just like the availability of a service. We cover this in data observability.

Lineage: where each data point comes from

Lineage documents the path data takes: which system it originates in, which processes transform it and which tables, reports or models it ends up in. In an AI pilot it serves three concrete purposes:

• Explaining an output. If the model produces an odd answer, lineage lets you trace which input data caused it and where that data came from.
• Assessing the impact of a change. If a source system changes a field's structure, lineage tells you which models are affected before they break.
• Answering an audit. Risk or an external auditor may ask you to show where the data used to train or feed the system came from.

Lineage documented by hand in a diagram ages quickly. It is better to capture it automatically from the processes that move the data. OpenLineage, an open project of the LF AI & Data Foundation, defines a standard for emitting lineage events built on three entities (datasets, jobs and runs), enriched with additional metadata called facets. Many orchestrators and processing engines have integrations with this standard, so you can start without rewriting your pipelines.

For the pilot, the reasonable minimum is being able to answer, for every data point that feeds the model, which source system it comes from and what transformations it went through. If you cannot answer that today, that is the first piece of work.

Permissions: who can use what

The fourth foundation is the one that stops the most pilots in regulated organizations. Having technical access to data does not mean having permission to use it in an AI system. There are three questions to answer:

Is there a basis for this use?

If the data includes personal information, using it in an AI system must be compatible with the purpose for which it was collected and with the applicable legal basis. In Spain the General Data Protection Regulation applies; in Latin America, each country has its own personal data protection law. The answer comes from the data protection officer or the legal team, not from the pilot team.

Do you need the data as it is?

Many use cases work just as well with pseudonymized or aggregated data, or with sensitive fields removed. Minimizing before you start reduces risk and simplifies approvals.

Who can access the output?

A model or an assistant can expose information from the data it works with. If an employee cannot see a piece of data in the source system, they should not be able to get it by asking the AI. The pilot's permissions must inherit those of the source. This ties into the broader discussion of the CISO's role in the face of AI.

Pre-pilot readiness checklist

We recommend reviewing this checklist with the data owner, Security and Compliance before approving the start of the pilot. If an item is not met, it does not mean the pilot cannot start, but it does mean that work goes into the plan with an owner.

• Catalog. Every dataset in the pilot has a record, a business owner and a technical custodian.
• Catalog. The key terms of the use case have an agreed, written definition.
• Quality. The relevant dimensions were chosen and thresholds were set before measuring.
• Quality. There is a baseline measurement and it is clear how it will be repeated if the pilot moves forward.
• Lineage. For every input, the source system and the transformations applied are known.
• Lineage. The data arrives through a repeatable process, not a manual extract.
• Permissions. The data is classified and it is known whether it includes personal data.
• Permissions. The basis for the use was validated by the data protection function or Legal.
• Permissions. Minimization was applied and access to the output respects the source permissions.
• Governance. The use case is registered in the organization's AI system inventory.

The last item links to the AI management system. ISO/IEC 42001 includes controls on the data used by AI systems, including its quality, provenance and preparation. If your organization already works with that standard, this checklist is a practical way to apply those controls to a specific case. We explain it in more detail in ISO/IEC 42001 in practice.

One final criterion: the first pilot should also be chosen for its data, not just its value. A use case with data that is already cataloged, has a single owner and contains no personal data is a better first step than a more ambitious one with scattered data. What you learn about preparing data carries over to every pilot that follows. Before using data with regulatory impact, validate the approach with Compliance, Risk and the data protection officer.