In this analysis
The error arrives before the model
A delivery report shows that a district’s performance has suddenly improved. The dashboard is functioning, the calculation is reproducible, and every row passes its type checks. Later, an analyst discovers that a source system stopped sending delayed deliveries. The remaining records are accurate descriptions of an incomplete population. This hypothetical failure is a reminder that data quality cannot be reduced to whether individual values look valid.
Wang and Strong’s research treats quality from the data consumer’s perspective, extending it beyond accuracy to dimensions such as relevance, completeness, representation, and accessibility. 1 The central issue is fitness for use. A dataset suitable for a historical summary may be unsuitable for an operational alert, even when both applications read the same table.
Validity is necessary and insufficient
Schelter and colleagues describe automated large-scale verification through declarative data-quality constraints. Great Expectations provides an implementation framework for defining and validating expectations about data. 28 These approaches can make checks repeatable and visible, but the checks still encode assumptions chosen by people.
A non-null delivery date does not establish that every delivery has arrived in the dataset. A positive transaction amount does not establish that refunds have been represented correctly. Our recommendation is to connect each check to a failure it is intended to detect. Where possible, compare counts and totals against an independent operational record. The goal is not to maximize the number of tests; it is to cover the ways a decision could become misleading.
Quality failures propagate through decisions
Sambasivan and colleagues’ qualitative research describes data cascades: downstream problems arising from upstream data issues and the organizational conditions around them. The study draws on interviews with practitioners in high-stakes AI, so its findings should not be interpreted as a population-wide failure rate. 4 Its contribution is to make incentives and coordination visible within a technical problem.
In the delivery example, an analyst might repair the report locally while the source-system team remains unaware of the missing events. The next model then trains on the same incomplete history. A correction that stays inside one notebook can leave the underlying failure intact. Treating quality as shared operational work changes who receives the incident and what counts as resolution.
Training data and serving data can disagree
Breck and colleagues’ data-validation work addresses production machine-learning concerns such as schema anomalies and differences between expected and observed data. Polyzotis and colleagues describe broader data-management challenges across production ML workflows. 35 These sources explain why a model’s input pipeline needs scrutiny beyond the initial training extract.
A feature may be available in a historical warehouse but arrive too late for a live decision. A categorical field may acquire new values. A default value may silently replace an unavailable measurement. Each change can preserve the technical shape of the input while changing its meaning. We recommend recording when each field becomes available and testing the actual serving path, rather than assuming it reproduces the training environment.
A dataset needs an account of its origins
Datasheets for Datasets proposes documentation covering motivation, composition, collection, and recommended uses. W3C’s PROV framework describes entities, activities, and agents involved in producing information. 610 Both make lineage and context inspectable, though they operate at different levels of abstraction.
For the delivery report, provenance should help trace a number back through transformations to source events and responsible systems. The useful question is not merely “Which table did this come from?” but “What happened to the population before it reached this table?” A chain of documented transformations can reveal where exclusions, deduplication, or late-arriving events changed the interpretation. It also gives future maintainers a starting point when the original analyst has moved on.
Quality measurements also need definitions
The W3C Data Quality Vocabulary supports describing quality dimensions, metrics, and measurements rather than treating quality as a single universal score. 7 A completeness metric is meaningful only when its expected population and measurement procedure are stated. Ninety-nine percent complete relative to received files says something different from ninety-nine percent complete relative to all business events.
Our proposed practice is to attach an owner, denominator, refresh cadence, and intended decision to each important quality measure. Record exceptions instead of silently normalizing them away. A low score may be acceptable for exploratory analysis and unacceptable for a regulatory submission. This is a judgment about use and consequence, not an invitation to change thresholds until the data pass.
The repair has to survive the next run
Xin and colleagues analyze production ML pipelines and identify opportunities arising from their execution patterns and repeated work. Their empirical setting is large-scale production infrastructure; it should not be read as a prescription that every small team adopt the same machinery. 9 It does reinforce the importance of considering recurring pipelines rather than one successful execution.
When a quality incident is repaired, replay affected periods where practical, check dependent outputs, and add a targeted regression check. Decide whether the consumer needs corrected historical data or simply a warning about a known limitation. A pipeline that becomes green while downstream decisions remain based on stale results has recovered technically without recovering informationally.
An invariant is a statement about the world
Suppose an import pipeline expects one record for every completed shipment. A uniqueness check on shipment identifiers can detect duplicates; it cannot discover identifiers that never arrived. Completeness requires an expected population, such as an independent completion ledger. Even that comparison needs aligned timing, because a legitimate reporting delay can resemble missing data.
This is why constraint-based verification must be tied to the meaning of the event, not only the shape of the table. 2 A useful invariant might state that every completed shipment becomes visible within an agreed delay, with exceptions explicitly recorded. Monitor both the invariant and its assumptions. If the independent ledger changes its definition of completion, the check may start measuring a different relationship. The technical test remains executable while its interpretation becomes wrong—a failure that demands ownership and documentation as well as code.
Make the responsibility visible
The operational model can be simple: producers own the meaning and delivery of source events; platform teams own reliable transformations; consumers state the conditions their decisions require. The exact division varies, but an incident should not fall into the gap between “the data arrived” and “the report ran.”
Scientific and engineering work on quality offers useful methods, but no automated framework can infer every business meaning. Our conclusion is that durable quality emerges from explicit definitions, repeatable checks, provenance, and a repair process that reaches the source. The successful dashboard is not the one that always looks orderly. It is the one whose users can discover what is missing, understand how it affects the decision, and know who can correct it.
Every important data check needs an owner, a business meaning, and a response.
Sources & methodology
This editorial synthesizes primary research, technical standards, and original engineering documentation. It is not an original experimental study or a systematic literature review. Research findings apply to the conditions studied; product recommendations are our interpretation. Figures explain concepts and do not present experimental measurements.
Sources checked on 31 October 2026. Publication dates and preprint versions may differ; the linked record identifies the referenced work. To suggest a correction, email contact@data-sorcerers.com.
- Wang and Strong (1996) — Beyond Accuracy: What Data Quality Means to Data Consumers ↗Back to text ↑
- Schelter et al. (2018) — Automating Large-Scale Data Quality Verification ↗Back to text ↑
- Breck et al. (2019) — Data Validation for Machine Learning ↗Back to text ↑
- Sambasivan et al. (2021) — “Everyone Wants to Do the Model Work, Not the Data Work”: Data Cascades in High-Stakes AI ↗Back to text ↑
- Polyzotis et al. (2017) — Data Management Challenges in Production Machine Learning ↗Back to text ↑
- Gebru et al. (2021) — Datasheets for Datasets ↗Back to text ↑
- W3C (2016) — Data on the Web Best Practices: Data Quality Vocabulary ↗Back to text ↑
- Great Expectations — GX Core Overview ↗Back to text ↑
- Xin et al. (2021) — Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities ↗Back to text ↑
- W3C (2013) — PROV-Overview ↗Back to text ↑
