# DearAuditor Eval

> Validation pipelines for clinical AI workflows. DearAuditor Eval builds repeatable, auditable evaluation pipelines that answer one question for clinics and clinical-AI vendors: does the AI still work on your patients? Each pipeline runs a fixed case bank through a perturbation grid, grades outputs with a pinned LLM judge (with a Cohen's kappa human-calibration gate built in, not yet exercised on these public examples), archives every run under a content hash, and emits a QMS-ready validation dossier. Powered by the open-source validrig engine (AGPL-3.0); source repositories open at launch.

Contact: aliaksei@dearauditor.ch. Sibling properties: https://qms.dearauditor.ch (open eQMS baseline), https://dearauditor.ch (auditing toolset).

## Pages

- [Front page](https://eval.dearauditor.ch/): what a validation pipeline is and why it matters
- [Methodology](https://eval.dearauditor.ch/methodology): packs, perturbation axes, judge calibration (and its current status), pinned runs, dossiers
- [Oncology tumor board preparation](https://eval.dearauditor.ch/tumor-board): An AI drafts the case summary a lung-cancer tumor board works from. This pipeline checks whether the draft survives missing documents, model swaps, and de-identification — on real, de-identified TCGA lung cases.
- [Telemedicine consultation triage](https://eval.dearauditor.ch/telemedicine-consultation): An AI works through a remote consultation — complaint, history, attachments — and proposes diagnosis, workup, and treatment. This pipeline grades three Gemini generations against gold answers on original synthetic consultation cases, and shows what an honest FAIL looks like.

## Repositories (open at launch)

- [validrig engine](https://github.com/AliakseiT/validrig): use-case-agnostic evaluation engine, `rig` CLI, AGPL-3.0
- [tcga-lung pack](https://github.com/AliakseiT/clinical-eval-tcga-lung): tumor-board pipeline on de-identified TCGA lung-cancer cases
- [consultation pack](https://github.com/AliakseiT/clinical-eval-consultation): telemedicine triage pipeline on original synthetic cases

## Notes for agents

- Findings on pipeline pages are generated from pinned runs by `rig publish`; the run hash appears next to each dossier.
- The telemedicine consultation pipeline includes a pre-registered acceptance gate that FAILED; it is published as-is — honest failures are the point.
- This site presents methodology and worked examples, not regulatory advice.