Measuring ADaM to SDTM traceability, from L0 to L3

Everyone agrees traceability matters. Almost no one can say how much of it a study has. It gets described in words, "traceability is good", "the derivations are documented", and those words are true right up until an auditor picks the one analysis variable whose path back to the raw data no one can reconstruct.

The reason it stays qualitative is that traceability is not one property, it is a property of every variable, and a study has hundreds of them. To put a number on it you have to grade each variable and then count. That is what r4subtrace, part of our open source R4SUB toolkit, does: it builds the derivation graph from the ADaM and SDTM metadata and assigns every analysis variable a traceability level.

The graph first

Before anything is graded, the variables and their declared sources become a directed graph. An ADaM variable points to the SDTM variables or predecessor ADaM variables it was built from, those point to their sources in turn, and the raw collection sits at the root. Most of the value is already visible once the graph exists. A variable with no edge leaving it is an orphan. A chain that stops at a dataset name rather than a variable is a dead end you can see.

Four levels, so the grade is a number

Each variable gets one of four levels. The definitions are deliberately plain, so two people grading the same study land on the same answer:

L0, no declared source. The variable appears with nothing to trace back to. L1, a source dataset is named, but not the specific variable or the method. L2, the specific source variable or variables are identified. L3, the derivation is fully specified, enough to reconstruct the value from the source without reading the code.

L3 is the same bar we set for a good method description in Define-XML as the dataset's contract: a reviewer can rebuild the value from the source data without opening a program. The levels simply make that bar countable across a whole study.

One headline indicator

The metric that summarises a study is TRACE_VAR_COVERAGE_L2PLUS, the share of analysis variables graded L2 or better. L2 is the point where a reviewer can follow a value to a named source rather than a folder, so L2-plus coverage is a fair line between traceability that holds up and traceability that only reads well. A study at 0.96 has a short, specific list of variables to fix. A study at 0.70 has a structural problem it should find now, not at the review.

Because it is a single share, it also trends. Run it on each dry run and the number should climb toward the filing, and a drop between runs is a regression worth a question. That is traceability behaving like every other quality metric, measured the same way each time.

Where coverage drops, predictably

Three places account for most of the L0 and L1 variables we see. Assigned values with no method, where origin says assigned and nothing explains the assignment. Multi-source derivations that name one input and quietly use two, so the graph is missing an edge. And carried-forward or imputed values whose method describes the intent but not the rule precise enough to reproduce. None of these fail a conformance check, because the datasets are internally consistent; they fail only when someone tries to walk the path, which is exactly what the level grading does on every run.

This is the quantitative companion to traceability from CRF to table, which argues for the practice; the levels are how you tell whether the practice is actually holding. The coverage number then feeds the Traceability pillar of the Submission Confidence Index, so a thin derivation chain lowers the readiness score directly, with the offending variables one click away.

The R4SUB packages

R4SUB is open source and MIT licensed. The trace graph and the L0 to L3 grading are implemented in r4subtrace (on CRAN), part of the R4SUB ecosystem (r-universe).