feat: eval package scaffold + CLI skeleton + OCAS schemas #69

New Issue

2026-06-04T15:13:19Z

xiaoju commented

2026-06-04 15:13:19 +00:00

Phase 1a of eval framework (#34)

Create @united-workforce/eval package with:

Package scaffold: packages/eval/, tsconfig, biome, package.json (bin: uwf-eval)
CLI skeleton with commander: run, report, diff, list subcommands (stubs)
OCAS schema registration:
- @uwf/eval-run — full eval execution record
- @uwf/eval-judge-frontmatter — frontmatter compliance data
- @uwf/eval-judge-upstream — upstream consumption data
- @uwf/eval-judge-hallucination — hallucination detection data
- @uwf/eval-judge-token-stats — token usage data
Type definitions: TaskManifest, JudgeInput, JudgeOutput<T>, EvalRun
task.yaml loader + validator

Acceptance Criteria

uwf-eval --help works
All schemas registered in OCAS
Types exported from package

Ref: #34

— 小橘 🍊（NEKO Team）

## Phase 1a of eval framework (#34) Create `@united-workforce/eval` package with: - [ ] Package scaffold: `packages/eval/`, tsconfig, biome, package.json (bin: `uwf-eval`) - [ ] CLI skeleton with commander: `run`, `report`, `diff`, `list` subcommands (stubs) - [ ] OCAS schema registration: - `@uwf/eval-run` — full eval execution record - `@uwf/eval-judge-frontmatter` — frontmatter compliance data - `@uwf/eval-judge-upstream` — upstream consumption data - `@uwf/eval-judge-hallucination` — hallucination detection data - `@uwf/eval-judge-token-stats` — token usage data - [ ] Type definitions: `TaskManifest`, `JudgeInput`, `JudgeOutput<T>`, `EvalRun` - [ ] task.yaml loader + validator ## Acceptance Criteria - `uwf-eval --help` works - All schemas registered in OCAS - Types exported from package Ref: #34 — 小橘 🍊（NEKO Team）

xiaoju referenced this issue

2026-06-04 15:13:20 +00:00

feat: uwf-eval run command — prepare, execute, collect #70

xiaoju referenced this issue

2026-06-04 15:13:23 +00:00

feat: builtin judges — frontmatter, upstream, hallucination, token-stats #71

xiaoju referenced this issue

2026-06-04 15:14:04 +00:00

test: E2E eval — 真实 agent 效果评估 #34

xiaoju referenced this issue from a commit

2026-06-04 23:42:24 +00:00

feat: eval package scaffold with CLI, schemas, types, task loader