Oncology tumor board preparation

The clinical task

Before a tumor board meets, someone has to condense each patient's chart into the brief the board actually discusses. Pathology report, clinical summary, molecular findings. This is exactly the kind of drafting work clinics now hand to an AI model.

The workflow under test: the model reads the case documents and produces a structured summary with diagnosis, stage, and the findings that drive treatment options.

What can go wrong

The dangerous failure is not a wrong answer. It is a confident answer built on missing evidence. If the pathology report never made it into the prompt (a broken upload, a permissions gap, a truncated context), a good model should say so. A bad one fills the gap with plausible text.

• Making things up when inputs are missing. The summary states a diagnosis and stage no source document supports.
• Silent drift on model updates. The vendor swaps the underlying model and yesterday's validated behavior quietly changes.
• De-identification gaps. Patient identifiers slip through the pipeline into prompts and logs.

How we measure it

The pipeline runs every case through a stress grid: the same case with individual documents removed, reordered, or degraded. A pinned AI judge scores each output against a rubric. The difference tells us how much each document actually contributes, its information value. A document the model can lose without its score dropping is a document the model is not really using.

The same setup re-runs whenever a model changes, so a swap becomes a measured check instead of a hope.

Findings

We ran the same pinned battery on two Gemini generations. The headline: how much a model leans on the pathology report depends on the model version. Which is exactly why the measurement has to be repeated on every swap.

Model-swap check: re-running the comparison battery, gemini-2.5-flash to gemini-3.5-flash showed a mean score delta of 0.00 (0 case-level regressions). The gate PASSED. That is the point: the swap was verified, not assumed.

How much weight these numbers carry: this case bank holds 8 cases. Enough to demonstrate the method and to catch a gross failure, far too few to certify a model. An information value of 0.08 versus 0.00 is a difference of two graded checks, well inside the noise a bank this size produces. A clinic engagement runs the same pipeline on hundreds of local cases, where the confidence intervals actually close. Read these figures as a worked demonstration, not as a verdict on either model.

De-identification check: adding clinical recognizers to the ingest pipeline raised measured identifier recall from 86% to 100% on synthetic identifiers injected into the case bank. 100% here is an upper bound. Recall is only measurable against identifiers we know are there, so it demonstrates the fix, not the absence of all leaks.

What it means for you

If your tumor-board assistant scores like this, the summary it hands your board can read perfectly while resting on nothing. Whether that happens can change with a routine model update, as it did between these two versions. The fix is not a better prompt. It is a contract: the workflow must refuse to summarize when a required document is absent, and your validation pipeline must prove it refuses.

That proof is repeatable. When the vendor ships a new model version, the same pack re-runs in minutes and tells you whether the contract still holds, for that version, not the last one.

The validation report

Every pipeline run ends in a dossier like this one — the document you hand to your quality manager or auditor. It is generated from the pinned run, never edited by hand.

generated from run 9d1cfd7827de6129