lemma × reflection · build with beam · verifiable rewards

The Grader

A deterministic reward function for agentic science audits. Paste an audit script — it is executed in a sandbox, SUMMARY_JSON is extracted, and the metrics are scored against a hand-verified answer key.

no LLM in the reward path
  1. 01run
  2. 02parse
  3. 03check
  4. 04score
specimen input task → script → stdout
Task · paper / claim
claim under audit
answer key · tolerance
Recorded attempts — click to load & grade
Audit script
audit.py stdout must contain  SUMMARY_JSON={…}
verdict reward ∈ [0, 1]
fetching python runtime…
every attempt preserved — crashes included · reward-hacking guards cap at 0.25 (metric == answer-key literal in source, or decisive metric with no computation) Papajams/beam-audit-bench · udirobert/lemma · evidence room