Turning Pinnacle 21 and Define-XML output into evidence you can score

Every study already produces the raw material for a readiness judgement. The Pinnacle 21 report lists conformance findings. The Define-XML file describes every variable, its origin and its codelist. The datasets carry the values. The problem is that these live in different formats and get read by different people, so no single view of the study exists that a machine, or a new reviewer, can read in one pass.

The fix is not another report. It is a common shape. In R4SUB, our open source and MIT licensed toolkit, that shape is a single evidence table, and everything else in the framework reads from it. This post is about the table, because the schema is the part worth standardising and the part other teams can adopt whether or not they use our code.

One row per finding, one fixed set of columns

The evidence table has a deliberately small, fixed schema. Every finding from every source becomes one row with the same columns:

run_id, study_id, asset_type, indicator_id, severity, result, metric_value.

run_id ties a row to the exact run that produced it, so a score is always reproducible back to its inputs. study_id and asset_type say what was examined, a dataset, the Define-XML, the reviewer's guide. indicator_id names the question being answered. severity ranks how much a failure matters. result is pass, fail or not applicable, and metric_value holds the number behind it, a percentage covered or a count, so the row carries the measurement, not just the verdict.

Because the columns are fixed and source-independent, a Define-XML finding and a Pinnacle 21 finding sit in the same table in the same shape. Quality stops being a folder of unlike reports and becomes one thing you can filter, count and score.

Filling the table: two parsers

Two functions do the ingestion. p21_to_evidence() reads a Pinnacle 21 validation export and maps each conformance finding to a row, carrying the rule identifier into indicator_id and the finding count into metric_value. Nothing about the study is re-judged here, the conformance engine has already done its work; the parser only puts the output into the common shape.

define_xml_to_evidence() parses the Define-XML itself and emits rows for the things the file is supposed to guarantee. A worked example: the indicator Q-DEFINE-001, label completeness. The parser walks every variable in the define file, checks that a non-empty label is present, and records the share that pass in metric_value. If 4 of 220 variables have an empty label, the row reads result fail, metric_value 0.98, and a reviewer can go straight to the four that need attention rather than reading the whole file.

This is the kind of check human review is worst at and automation is best at, a comparison across a long, almost-complete list. We made the same argument about codelist drift in Define-XML as the dataset's contract; the evidence table is where that argument becomes a row that scores on every run.

Why the shared schema is the point

Once findings share a schema, three things follow. Indicators can be written once and applied to any source that fills the table, so a new check is a new indicator, not a new pipeline. The same evidence can be scored against different weightings without regathering it, which is what lets one study be judged for more than one regulator. And the score drills all the way down: a low Quality pillar opens to the exact rows, and each row names the asset and the metric behind it.

The schema is also deliberately domain-agnostic. Nothing in those seven columns is specific to clinical data, which is by design, but for a submission the payoff is immediate: the conformance work you already run and the Define-XML you already write become inputs to a readiness score rather than artefacts filed and forgotten.

The evidence table feeds the four pillars and the single index described in submission readiness as a number, not a meeting, and it draws on the mapping and traceability practices in the SDTM to ADaM walkthrough. Start with the table. Everything else is a function of it.

The R4SUB packages

R4SUB is open source and MIT licensed. The evidence schema and the two parsers described here live in r4subcore (on CRAN), part of the R4SUB ecosystem (r-universe).