Notes from the review council

The idea

Three fixed models read an evidence pack built from the repo: measured facts such as code size, tests and commit history, the team's deck, committed decks and docs, the demo page text and descriptions of its screenshots — at most about 75k tokens. Each model scores every official criterion of the task from 0 to 10; the median counts, the task's official weights turn it into a score out of 100, and a fourth model writes the review. The council never runs the code, never sees the pitch, and for finalists is never told how the project placed.

The full design — the prompts, the per-task rubrics and reviewing guides built from the official rules, the queue that gives every project the identical panel — is in the repo.

What the measurements said

Before the council, every project with public code had a blind review by one strong model. The council was measured against those blind reviews — mean absolute gap per project, and how often the top-ranked project was the same:

In every task with several reviewed projects, the council's top project was also the blind review's top project. It was never the jury's winner: in the five tasks whose winner was reviewed, the winners scored 57.5–69 (average 63.7, solidly strong) while the council's top project scored 66–83. The other five winners were never reviewed — four had no public repo found by the search, and one was deleted or made private after the event.

That gap is the finding, not a defect: the jury also had mentors' notes, the paper round and the live pitch; the council read only what was public. A high council score means strong public work, not that the jury was wrong.

What broke

• One member's upstream provider answers HTTP 400 to any request containing a jailbreak phrase — and evidence packs deliberately contain them, because teams test prompt injection. Jailbreak strings are now masked everywhere, a refused request is resent up to four times, and only then does the member read a pack without quoted code.
• Free-model quotas and busy spells shaped the queue: rate-limited members are waited for with a growing backoff, a used-up plan window is retried every 30 minutes, and the panel never shrinks — every project faces the identical three members.
• The prompt-injection scan first flagged any mention of a system prompt, putting false "tries to steer the reviewers" flags on five projects; it now flags score-steering anywhere, and jailbreak attempts only in text the team itself wrote.
• Deploys overlap for about a minute with two workers on one SQLite database; jobs are claimed atomically and a job with no progress for 20 minutes goes back to the queue.

How it ended

The form was open for the event week, 4–11 October 2026: 62 reviews in total — 21 finalists and 41 projects teams sent in themselves. When the queue drained, the form closed and the site became this static archive; the repo keeps the council, the evidence builder, the prompts and every review.

If your project was reviewed, the number was written by the panel above, the same one every other project faced. Thanks to every team that sent its work in.