Data observability: freshness, volume, schema and lineage

Operations teams know when an application goes down: monitoring detects it and someone gets an alert. Data is different. A pipeline can finish "successfully" and still leave a table with half its rows, with yesterday's data or with a column that changed type. Nobody gets an alert. The business finds out when a dashboard shows an odd number or, worse, when a model or a regulatory report has already used that data.

Data observability applies to pipelines the same discipline we apply to services: defined signals, measurable objectives, alerts with owners and incident management. It does not require a specific platform to get started. It requires deciding which data matters, what "good" means for it and who responds when it is not.

A pipeline is a service

For whoever consumes it, a data set has the same properties as any service: it must be available when needed, be correct and behave predictably. The difference is that data failures are rarely loud. A service that is down returns errors; an incomplete table returns plausible results.

That is why job status (finished or failed) is not enough. It is the equivalent of checking that a server is powered on without checking whether it responds correctly. For applications we distinguish which question each signal answers, as we explain in logs, metrics and traces. Data calls for the same approach: separate the signals and measure each one where it adds value.

The five operational signals

• Freshness. When the data was last updated compared with what the consumer expects. It is the simplest signal and the one that catches the most failures: loads that did not run, ran late or processed an old file.
• Volume. How many rows or records arrived, compared with what is normal for that day and time. A sharp drop usually points to an incomplete source; a spike, to duplicates.
• Schema. Columns added, removed, renamed or changed in type. Many data incidents start with a legitimate change in the source system that nobody communicated to the teams consuming that data.
• Distribution. Whether values are reasonable: share of nulls, out-of-range values, new categories, key uniqueness. It catches errors that change neither volume nor schema.
• Lineage. Which tables and processes depend on which. It is not an alert in itself, but it turns an alert into a diagnosis: it shows where the problem comes from and whom it affects. OpenLineage defines an open standard to capture it, with a model of jobs, runs and data sets that emit events every time a process runs.

What to measure at each pipeline stage

Not every signal is measured everywhere. Measuring each one where it adds value avoids duplicate alerts and shortens diagnosis.

At the source, the goal is to catch schema changes and abnormal volumes early, before they enter the platform. At ingestion, the freshness of each load and the comparison between rows read and rows written. During transformation, schema and distribution tests on the resulting models. At consumption, freshness and volume as the dashboard or model sees them, which is what actually matters to the user. When a test fails at consumption, lineage shows which load or source to check first.

You do not need to cover the whole pipeline to get started. We recommend choosing a limited number of critical data sets (for example, five to ten), such as those feeding regulatory reports or customer processes, and walking their lineage back to the source. That walk shows which loads and sources to watch first, who owns each segment and where tests are missing. It is a modest inventory, but it turns an abstract discussion about data quality into a concrete list of signals, thresholds and owners.

Signals, what they catch and example thresholds

The thresholds in this table are illustrative. Each data set needs its own, agreed with whoever consumes it.

Signal	What it catches	Example threshold (illustrative)
Freshness	Loads that did not run or arrived late	Warning if the last load is more than 2 hours old; error after 6
Volume	Incomplete sources or duplicates	Alert if the day's rows deviate more than 30% from the average for the same weekday over the last four weeks
Schema	New, removed or retyped columns	Any unregistered change to a critical table
Distribution	Nulls, out-of-range values, repeated keys	More than 1% nulls in a mandatory column; any duplicate key
Lineage	Origin of a failure and downstream impact	Every critical table with its dependencies captured automatically

Fixed thresholds are a good start because they are easy to understand and discuss. Anomaly detection on volume or distribution can come later, under the same conditions of history and accuracy measurement we describe in AIOps without the hype.

Start with tests next to the code

If your transformations use dbt, you already have two tools at hand: source freshness checks and generic column tests. The following example declares a source with freshness thresholds and uniqueness and not-null tests.

# models/sources.yml
version: 2

sources:
  - name: sales                     # raw data landed by ingestion
    schema: raw
    loaded_at_field: _loaded_at     # column with the load timestamp
    freshness:                      # illustrative thresholds
      warn_after: {count: 2, period: hour}
      error_after: {count: 6, period: hour}
    tables:
      - name: orders
        columns:
          - name: order_id
            tests:
              - unique              # no duplicate keys
              - not_null            # no empty keys

The dbt source freshness command evaluates freshness and dbt test runs the tests. Both can run in the orchestrator after each load and send their results to the alerting system. Great Expectations offers a similar approach in Python, with declarative expectations on volume, ranges and distributions and with readable reports for the team. The tool matters less than the habit: every critical table with its tests versioned alongside the code that produces it.

Data SLOs, alerts and owners

Tests answer "is it fine right now?". SLOs answer "does it meet what we promised?". The "Service Level Objectives" chapter of Google's SRE book defines an SLO as a target value for a measured service level indicator, and the idea carries straight over to data. An illustrative example: "the daily balances table is available before 7:00 a.m. on 99% of business days in the quarter".

We recommend:

• SLOs only for critical data sets: those that feed regulatory reports, customer processes or daily decisions.
• Indicators from the consumer's point of view, such as freshness at the point of consumption or completeness, not internal pipeline details.
• Alerts that interrupt someone only when the SLO is at risk; everything else goes to a queue reviewed during business hours. It is the same criterion we describe in alerts without fatigue.
• One owner per data set, recorded and visible, who responds to the alert and agrees thresholds with consumers.

Data incidents

When an SLO is breached or a critical test fails, treat it as an incident, with the same process as application incidents: logging, severity, communication, resolution and post-incident review. Severity should reflect the impact on consumers, not the technical complexity of the failure: a column full of nulls in a table that feeds a regulatory report can be more serious than a failed load in a test environment. On top of that, there are two particularities.

The first is containment. Besides fixing the pipeline, you have to decide whether to stop the processes that consume the affected data and notify those who already used it. Lineage answers that question in minutes; without it, the answer comes from asking around by email.

The second is backward correction. Fixing today's load does not correct previous days. Closing the incident must include reprocessing the affected periods and verifying that the tests pass on them.

Post-incident reviews feed the tests: every data incident should end with at least one new test that would have caught it earlier. And if the data comes from in-house applications, part of the fix lies at the source: recording schema changes and publishing load events, just as we ask for the software to be observable and not just the platform, as we explain in an observable platform is not an observable application.