The serious version

Pelly's Clinic is a pedagogical demo of Jetty's verification architecture. Under the pixel art, every mechanic is a production evaluation practice — the same ones you would stand up to ship an agent with confidence. This page is the mapping, with sources.

The premise

Evaluation platforms typically ask users to bring their own definition of quality — write scorers, curate datasets, design rubrics — before delivering value. Most agent builders don't have that process maturity, so most agents ship on vibes. Jetty's position is that a runbook already encodes a definition of quality: its steps and expected outcomes are a rubric the author wrote without knowing it. Evaluation scaffolding can therefore be derived rather than authored, and improvement ("hill climbing") can be experienced rather than configured. The game exists to make that loop — run, judge, fix, re-run, climb — something you feel in twenty minutes rather than read about in a whitepaper.

Academic grounding

Agent-as-a-Judge: Evaluate Agents with Agents Zhuge et al., 2024 · arXiv:2410.10934 · DevAI benchmark The core thesis: agentic systems that judge intermediate steps of a trajectory outperform outcome-only scoring. On DevAI (55 realistic AI-development tasks, 365 hierarchical requirements), Agent-as-a-Judge agreed with human expert consensus ~90% of the time versus ~70% for LLM-as-a-judge. In the game: tiles resolve per step, not per run; the judge annotates each step's artifact; the claim-extraction table is the judge's primary input. The artifact mechanic is intermediate-trajectory judging.

Survey on Evaluation of LLM-based Agents Yehudai et al., 2025 · arXiv:2503.16416 Systematizes agent evaluation across fundamental capabilities (planning, tool use, self-reflection, memory), application-specific benchmarks, and frameworks, and documents the field's shift toward continuous, fine-grained, trajectory-level evaluation. In the game: the four judge dimensions (correctness, conciseness, completeness, grounding) are a deliberately small rubric taxonomy; the clinic’s records view (The Hill, run log, rubric inventory) is the "continuous evaluation" surface the survey calls for, at toy scale.

Reflexion: Language Agents with Verbal Reinforcement Learning Shinn et al., 2023 · NeurIPS 2023 · arXiv:2303.11366 Agents improve across episodes by converting evaluation feedback into verbal guidance carried into the next attempt — no weight updates required. In the game: the dungeon loop is Reflexion made tactile: read the judge's note, convert it into a runbook patch (linguistic feedback), re-run the failed step. The patch persisting into runbook v1.1.0 is the episodic memory.

Industry practice

Eval-gated iteration. The runbook's pass threshold (≥ 4.0 overall, no criterion below 3) with a bounded fix loop mirrors eval-gated CI/CD: quality gates that block "ship it" until scores clear the bar.
Traces become regression tests. Gilbert's fish story is caught once in a trace, then converted into a standing check that runs on every future visit — the production-trace-to-regression-test workflow.
Rubric refinement. Run 3's authoring moment is the rubric-refinement loop: humans codify judgments the judge then enforces, and the rubric inventory compounds per failure type encountered.

The mapping

Game mechanic	Production concept
Overworld map of tiles	Runbook steps / execution trajectory
Tile resolves ✓ green / ✗ red	Step-level pass/fail attribution
Dungeon view (expected vs. actual + judge note)	Trace inspection with judge annotations
Candidate fixes → re-run one step	Targeted iteration and replay
Collectible artifacts per step	Show-your-work intermediary artifacts; absence is visible
Rubric checks as inventory	Rubric curation, one check per failure type
Party members (Pelly × model)	Quality lives in the runbook × model pair, not the model alone
Health bar	Aggregate reliability score across runs
The Hill	Score-over-runs with runbook-version flags — the hill-climb chart
Fish stories	Fabrication / grounding against ground-truth records

Why an ambient scribe?

Healthcare documentation has real quality stakes and buyers who need receipts: every claim in a note should trace to a line in the encounter and survive a check against the record. That is exactly the shape of Jetty's verification story — runbooks whose steps declare expected artifacts, judged trajectories, and machine-readable validation reports. The pelican simply makes the compliance vibe survivable for a demo. The data structures in the game (runbook frontmatter, step manifests, validation_report.json) follow Jetty's published runbook format, so what a player learns transfers directly.

Play Pelly's Clinic →