How a validation pipeline works

The pack: a validation pipeline as files

A pack is a complete description of one validation pipeline: the clinical task, the case bank, the grading rubric, the distortions to apply, and the thresholds for passing. It is a folder of seven YAML files plus cases. No code.

That matters for two reasons. A pack can be reviewed by a clinician or a quality manager the way they review any controlled document, line by line. And a pack can be re-run: the same files produce the same evaluation next month, on the next model version, at the next audit.

The engine that runs packs does not know what a clinical task is. The clinical knowledge lives in the pack; the machinery lives in validrig.

The case bank

Every pack runs on a fixed set of cases with gold answers: what a correct output must contain, decided before any model runs. Our public examples use open or synthetic data only: real de-identified TCGA lung-cancer records for the tumor board pipeline, and original synthetic consultations for the telemedicine pipeline. Clinic engagements build case banks from the clinic's own cases, and that data never leaves the clinic.

Stress-testing the inputs

A model that answers well on the clean case has told you little. The pipeline runs each case through a grid of controlled distortions and measures what changes:

• Withholding documents removes one source document at a time. If scores do not drop when a document disappears, the model was not using it. It filled the gap with plausible text. We call the score drop the document's information value. An information value near zero on a critical document is a finding, not a compliment.
• Reordering and reformatting shuffles document order and degrades formatting. A robust workflow should not care.
• Swapping the model runs the same grid on a new model version and compares the scores. An update becomes a measured check instead of a guess.

The judge, and why it needs checking

Scoring thousands of outputs by hand does not scale, so a second AI model grades each output against the rubric. The technique is called LLM-as-judge, run here as pinned G-Eval prompts. That raises the obvious question: why trust an AI to grade an AI?

You should not, until it is calibrated. Calibration means clinicians grade a sample of the same outputs with the same rubric, independently and in duplicate. The engine computes Cohen's κ: a chance-corrected measure of how often the judge agrees with the human graders beyond what luck would produce. The engine picks the outputs to grade, records the human grades, computes κ, and gates on it. When the rubric or judge prompt changes, calibration starts over.

On these two examples, that calibration has not been done. No clinician has graded these outputs. Every dossier on this site reports its calibration status as not_collected, and you can read that in the dossiers themselves. The machinery is built and the gate is real, but the human grades are missing. Treat the scores here as a demonstration of the method, not as a validated judgement of either model. Producing those grades with clinical experts is the first thing a real engagement does.

Frozen runs

Every run is frozen: the case bank, distortion grid, judge prompts, model identifiers, and every raw response are content-hashed into a single run identifier. Two consequences:

• Any number we publish links to one hash. If a page here says 0.21, there is an archived run in which 0.21 was measured, with every input and raw response kept. What freezing does not give you: a live LLM judge is not deterministic, so running the same pack again can move a score slightly. Runs are traceable and auditable rather than bit-for-bit reproducible, and each run records that fact.
• The pages for the worked examples are generated from the frozen runs by the engine's rig publish command, the same data path that produces the dossiers. The site cannot drift from the evidence, because it is built from it.

The dossier

The end product of a pipeline is a validation dossier: a fixed-format report with scope, results, thresholds, judge calibration, and full traceability to the frozen run. It is written for a quality management system, not for a leaderboard. It is the document a clinic files as V&V evidence and shows an auditor. Each worked example on this site ends with its dossier rendered in full.

Source

The engine and the public example packs are open source (validrig is AGPL-3.0). The source repositories open at launch:

• github.com/AliakseiT/validrig — the engine and rig CLI
• github.com/AliakseiT/clinical-eval-tcga-lung — the tumor-board pack
• github.com/AliakseiT/clinical-eval-consultation — the telemedicine consultation pack