How a validation pipeline works

The pack: a validation pipeline as files

A pack is a complete, declarative description of one validation pipeline: the clinical task, the case bank, the grading rubric, the perturbations to apply, and the thresholds that count as passing. It is a folder of seven YAML files plus cases — no code.

That matters for two reasons. A pack can be reviewed by a clinician or a quality manager the way they review any controlled document, line by line. And a pack can be re-run: the same files produce the same evaluation next month, on the next model version, at the next audit.

The engine that executes packs is use-case-agnostic. The clinical knowledge lives in the pack; the machinery lives in validrig.

The case bank

Every pack runs on a fixed bank of cases with gold answers — what a correct output must contain, decided before any model runs. Our public examples use open or synthetic data only: real de-identified TCGA lung-cancer records for the tumor board pipeline, and original synthetic consultations for the telemedicine pipeline. Clinic engagements build case banks from the clinic's own workflows, and that data never becomes public.

Perturbation axes

A model that answers well on the clean case has told you little. The pipeline therefore runs each case through a grid of controlled distortions and measures what changes:

• Input contract — withhold one source document at a time. If scores do not drop when a document disappears, the model was not using it: it fills the gap with plausible text. We call the score drop the document's information value, and an information value near zero on a critical document is a finding, not a compliment.
• Order and format — shuffle document order, degrade formatting. Robust workflows should not care.
• Model swap — run the identical grid on a new model version and diff the scores. An update becomes a measured regression check instead of a leap of faith.

The judge — and why the judge needs checking

Scoring thousands of perturbed outputs by hand does not scale, so a second AI model grades each output against the rubric (the technique known as LLM-as-judge, run here as pinned G-Eval prompts). That immediately raises the right objection: why trust an AI to grade an AI?

You should not — until it is calibrated. Calibration means clinicians grade a sample of the same outputs with the same rubric, independently and in duplicate, and the engine computes Cohen's κ: a chance-corrected measure of how often the judge agrees with the human graders beyond what agreement by luck would produce. The engine samples the outputs to be graded, records the human grades, computes κ, and gates on it — and when the rubric or judge prompt changes, calibration starts over.

On these two examples, that calibration has not been done. No clinician has graded these outputs; every dossier on this site reports its calibration status as not_collected, and you can read that in the dossiers themselves. The machinery is built and the gate is real, but the human grades are missing — so treat the scores here as a demonstration of the method, not as a validated judgement of either model. Producing those grades with clinical experts is the first thing a real engagement does.

Pinned deterministic runs

Every run is pinned: the case bank, perturbation grid, judge prompts, model identifiers, and every raw response are content-hashed into a single run identifier. Two consequences:

• Any number we publish traces to one hash. If a page here says 0.21, there is an archived run in which 0.21 was measured, with every input and raw response kept. Note what pinning does not give you: a live LLM judge is not deterministic, so re-running the same pinned pack can move a score slightly. Runs are traceable and auditable rather than bit-for-bit reproducible, and each run records that fact.
• The pages for the worked examples are generated from the pinned runs by the engine's rig publish command — the same data path that produces the dossiers. The site cannot honestly drift from the evidence, because it is built from it.

The dossier

The end product of a pipeline is a validation dossier: a fixed-format report with scope, results, thresholds, judge calibration, and full traceability to the pinned run. It is written for a quality management system, not for a leaderboard — the document a clinic files as V&V evidence and shows an auditor. Each worked example on this site ends with its dossier rendered in full.

Source

The engine and the public example packs are open source (validrig is AGPL-3.0). The source repositories open at launch:

• github.com/AliakseiT/validrig — the engine and rig CLI
• github.com/AliakseiT/clinical-eval-tcga-lung — the tumor-board pack
• github.com/AliakseiT/clinical-eval-consultation — the telemedicine consultation pack