Machine-readable release
EVALLM for the agents that build these systems
Medical AI tools are now routinely developed, tested and documented with the help of AI coding and research agents, and guidance that exists only as journal prose is guidance those agents apply inconsistently or not at all. EVALLM is therefore published in a structured machine-readable format alongside the human-readable framework.
The framework, in one document
evallm.json
All 84 statements with their domain, thematic group, actor, lifecycle stage, applicability, mapped standards, agreement percentage, round and consensus category — plus the eight domains, the controlled vocabularies each field draws from, the consensus thresholds, and all 28 adoption-checklist questions.
Instructions for an automated assessment
evallm-agent-instructions.md
How an agent should scope an assessment, select the applicable statements, judge each one, and report the result — including the rule that a verdict without a citation is unknown rather than met, and that silence where a study would have had to speak is not met rather than unknown.
Fetch both files
Both are served as static files over HTTP, so an agent can retrieve them directly. Version 1.0.0.
curl -O https://evallm.kather.ai/evallm.json
curl -O https://evallm.kather.ai/evallm-agent-instructions.mdThe shape of the data
Each statement is published exactly as it was rated, with the classifications the writing group applied to aid navigation. Those classifications do not rank statements and carried no weight in the consensus process.
{
"id": "2.1",
"domain": 2,
"group": "Error and harm measurement",
"statement": "Evaluation studies should report the hallucination rate, defined as …",
"actors": ["Evaluator"],
"stage": "Pre-deployment",
"appliesTo": "All systems",
"mapsTo": ["TRIPOD-LLM"],
"agreement": 100,
"round": 1,
"category": "Strong consensus"
}Working by hand instead?
The interactive checklist produces the same assessment as a PDF for a committee, a spreadsheet for analysis, or JSON you can reload later.