Oncology tumor board preparation

The clinical task

Before a tumor board meets, someone has to condense each patient's chart — pathology report, clinical summary, molecular findings — into the brief the board actually discusses. It is exactly the kind of drafting work clinics now hand to an AI model.

The workflow under test: the model reads the case documents and produces a structured summary with diagnosis, stage, and the findings that drive treatment options.

What can go wrong

The dangerous failure is not a wrong answer — it is a confident answer built on missing evidence. If the pathology report never made it into the prompt (a broken upload, a permissions gap, a truncated context), a good model should say so. A bad one fills the gap with plausible text.

• Confabulation under missing inputs — the summary states a diagnosis and stage no source document supports.
• Silent drift on model updates — the vendor swaps the underlying model and yesterday's validated behavior quietly changes.
• De-identification gaps — patient identifiers slip through the pipeline into prompts and logs.

How we measure it

The pipeline runs every case through a perturbation grid: the same case with individual documents withheld, reordered, or degraded. A pinned AI judge scores each output against a rubric, and the difference tells us how much each document actually contributes — its information value. A document the model can lose without its score dropping is a document the model is not really using.

The same pinned setup re-runs whenever a model changes, so a swap becomes a measured regression check instead of a hope.

Findings

We ran the same pinned contract battery on two Gemini generations. The headline: how much a model leans on the pathology report depends on the model version — which is exactly why the measurement has to be repeated on every swap.

Model-swap regression: re-running the pinned comparison battery, gemini-2.5-flash → gemini-3.5-flash showed a mean score delta of 0.00 (0 case-level regressions) — the gate PASSED. That is the point: the swap was verified, not assumed.

How much weight these numbers carry: this case bank holds 8 cases. That is enough to demonstrate the method and to catch a gross failure, and far too few to certify a model — an information value of 0.08 versus 0.00 is a difference of two graded checks, well inside the noise a bank this size produces. A clinic engagement runs the same pipeline on hundreds of local cases, where the confidence intervals actually close. Read these figures as a worked demonstration, not as a verdict on either model.

De-identification check: adding clinical recognizers to the ingest pipeline raised measured identifier recall from 86% to 100% on synthetic identifiers injected into the case bank. 100% here is an upper bound — recall is only measurable against identifiers we know are there, so it demonstrates the fix, not the absence of all leaks.

What it means for you

If your tumor-board assistant scores like this, the summary it hands your board can read perfectly while resting on nothing — and whether that happens can change with a routine model update, as it did between these two versions. The fix is not a better prompt — it is a contract: the workflow must refuse to summarize when a required document is absent, and your validation pipeline must prove it refuses.

That proof is repeatable. When the vendor ships a new model version, the same pinned pack re-runs in minutes and tells you whether the contract still holds — for that version, not the last one.

The validation report

Every pipeline run ends in a dossier like this one — the document you hand to your quality manager or auditor. It is generated from the pinned run, never edited by hand.

generated from run 9d1cfd7827de6129