Why a unified schema is only the beginning
A knowledgeable researcher can work reliably with analysis-ready data because they bring context. They understand the subject, recognise familiar structures and know when something does not look right.
A machine does not have that advantage. If it is expected to use data reliably, the meaning, relationships and limitations of that data need to be explicit enough for the intended use.
That distinction matters because “AI-ready” is increasingly used to describe data that has been moved to a modern platform, given standard column names or made available through a common query layer. All of these are worthwhile improvements. But they say more about accessibility and consistency than they do about whether the data can support a particular AI application.
The appeal of a unified model
When several programmes collect related data in different structures, a unified data model has an obvious appeal. Common identifiers, consistent names and a single interface can make a fragmented data estate much easier to navigate.
The difficulty is that “unified” can describe quite different levels of alignment. It may simply mean that sources share a technical structure or that tables and fields have been renamed and brought together.
At the other end, the underlying concepts, values and relationships have been reconciled so that the same thing really does mean the same thing across sources. That goes beyond having a common structure or matching field names.
I came across this in a recent review of two source mappings within a proposed unified schema. Across several hundred mapping rows and more than a dozen views, the field names were identical in both sources. At that level, the two mappings appeared well aligned.
Around one in five rows had a different human-readable label or recorded data type. The differences were in the mapping metadata, so they do not necessarily mean that the underlying fields were defined differently. Even so, the metadata was not consistent across the two mappings.
We then compared one of the mappings with its own release data dictionary and found further discrepancies. One field, for example, appeared as an integer in the source definition but as a date in the mapping.
Coverage told another part of the story. In one shared clinical domain, fewer than one in ten typed source fields appeared as exact matches at both view and field level. Several important clinical entities were largely outside the mapping, as was most of a linked administrative-data domain.
What we had was useful structural alignment across a selected set of fields. What we did not have was evidence that the wider source models were equivalent.
Structural harmonisation can remove a great deal of friction. Problems arise when we expect it to demonstrate something it was never designed to demonstrate.
What analysis-ready data should provide
Analysis-ready data should allow a knowledgeable researcher to work reliably without having to reconstruct the data pipeline first.
That does not necessarily mean hiding the source model. Quite often, preserving its structure helps.
Participant, referral, condition, observation and sample data, for example, may be easier to understand when they remain separate and source-aligned. The important thing is knowing what each table contains, how records are identified and how the tables connect. But a clear structure does not necessarily mean the data itself is reliable. Types, dates and permitted values should be verified against the records, rather than assumed to be correct because they appear in a data dictionary. The same applies to completeness, duplicate records and the links between tables. Each release also needs documentation that reflects what was actually published, including its limitations and enough information to reproduce it.
A researcher may still need specialist knowledge to select a cohort, interpret a code or decide whether a result makes sense. That is part of analytical work.
What changes for AI
Once data is being prepared for an AI application, some of the assumptions we can safely make about human users disappear.
A researcher can read documentation and decide how two fields relate. They can notice that the same clinical concept has been represented differently in two programmes. They can investigate whether a blank means “not recorded”, “not applicable” or “no known condition”.
An automated system needs those distinctions to be represented consistently enough for the task it has been given.
That puts more weight on entity resolution, relationships and time. Codes, units and categories may need to be normalised across sources. The distinction between missing and genuinely absent information needs to be preserved. Provenance needs to be detailed enough to trace a result back through its transformations. Data quality also needs to be measured over time, particularly when the data feeding a system can change.
Preparing data for AI means making more of the researcher’s interpretation explicit in the data and metadata.
Ready for what?
This is also why there is no universal AI-ready state.
A dataset can only really be ready for something.
Predictive modelling brings its own requirements. The population needs to be appropriate for the model, outcomes and labels need to be trustworthy, and the risk of leakage needs to be understood. It also matters whether changes in the incoming data can be spotted over time. Retrieval and AI-assisted search depend more heavily on metadata, permissions and provenance, particularly when users need to trace an answer back to its source. A dataset that works well in one setting may therefore need quite different preparation for another.
Table shape is not the deciding factor
This also makes the familiar debate about wide and narrow tables less useful than it first appears.
A wide table is not inherently difficult for a machine to use. If its grain, entity identifiers, relationships, time context and provenance are clear, it may be perfectly suitable.
A set of carefully structured narrow tables can still be troublesome if the joins are ambiguous or depend on knowledge that exists only in documentation or in the heads of experienced users.
Take a participant view assembled from consent, registration, family history and pipeline-status data. Its width is not the interesting part. The issue is whether enough of the original context survives the transformation to identify the entity, understand when the information applied and trace it back through any transformations to the source.
A researcher can often reconstruct those answers. The less we want a system to rely on that human reconstruction, the more explicitly the data product needs to provide them.
A layered approach
In practice, I would start by making the source-aligned data genuinely usable: preserve useful domains, assure the data, stabilise identifiers and make the joins clear. Harmonisation can then be added where there is a real cross-programme need, such as cohort discovery or longitudinal research.
The requirements for AI come from the application itself. Once that application is defined, it becomes possible to decide what additional semantics, lineage, quality thresholds, access controls or monitoring are needed. There is little value in building all of this in advance simply to attach an AI-ready label to the data.
This means data does not have to wait for a perfect common model before it becomes useful. Data does not have to wait for a perfect common model before it becomes useful. But a partially harmonised schema is not, by itself, evidence that the data is ready for AI.
The decision hidden inside the label
Before describing a dataset as AI-ready, I would want to know what it is expected to support and how we know it can do that reliably.
That question changes how the data is designed. Good schema coverage is useful, but only if the concepts and relationships researchers actually need are represented properly. Documentation and provenance also need to be built into the data product, so that results can be understood and traced back to their source. Technical alignment can then be judged against real research or product use cases, rather than taken as evidence that the underlying meaning is consistent.
Analysis-ready and AI-ready are useful distinctions when they describe what the data can actually support, rather than the technology used to provide it. A unified schema can help, but readiness is about more than bringing data into a common structure. Matching column names is useful, but the harder work lies in making sure the underlying meaning, context and relationships are consistent.
Evidence note: The examples in this article come from two reviews: a comparison of two source mappings within a proposed unified schema, and a comparison between one of those mappings and its release data dictionary. I have generalised the figures and removed field, table, programme, release and platform names. The review covered schema metadata only, not participant-level data.
Leave a Reply