xiaoju
a08775896f
fix: frontmatter judge handles parsed object output
...
CI / check (pull_request) Successful in 2m38s
The extract pipeline stores step output as a JSON object in CAS,
but the frontmatter judge only checked for raw markdown strings.
Now accepts both formats: parsed objects check $status directly,
raw strings go through YAML frontmatter extraction.
Fixes eval frontmatter-compliance scoring 0 on valid outputs.
2026-06-05 02:55:58 +00:00
xiaoju
8c26f16716
feat: builtin judges — frontmatter + token-stats (deterministic) + upstream/hallucination (stubs)
...
CI / check (pull_request) Successful in 1m45s
Implement 4 builtin judges for eval framework:
- frontmatter-compliance: validates YAML frontmatter with $status field,
score = stepsValid / stepsTotal
- token-stats: aggregates Usage from step nodes, always score 1.0
(informational only)
- upstream-consumption: LLM-as-judge stub (score 0, TODO)
- hallucination: LLM-as-judge stub (score 0, TODO)
Infrastructure:
- judge/builtin/read-steps.ts — shell out to uwf step list
- judge/builtin/types.ts — BuiltinJudge, BuiltinJudgeOutput
- runner/collect.ts — dispatch builtin judges by name
9 new tests (frontmatter validation + token aggregation)
Refs #71
2026-06-05 00:09:06 +00:00
xiaoju