feat: validate model connectivity during uwf setup

Send a test completion request after configuration to verify the model
is reachable. If validation fails, warn the user and suggest trying a
different model or checking their settings.

Fixes #335

.plan/335-setup-validate.md

@@ -0,0 +1,83 @@
+# Test Spec: uwf setup model connectivity validation (#335)
+
+## Context
+
+File: `packages/cli-workflow/src/commands/setup.ts`
+Test file: `packages/cli-workflow/src/__tests__/setup-validate.test.ts`
+
+After `cmdSetup` writes config, it should send a test chat completion request to verify the configured model is reachable. If validation fails, warn the user (don't abort — config is already saved).
+
+## Implementation Notes
+
+- Add a `validateModel(baseUrl, apiKey, model)` function that sends a minimal chat completion request (`POST /chat/completions` with `messages: [{role:"user",content:"hi"}]`, `max_tokens: 1`)
+- Returns `Result<void, string>` — ok if 2xx response, error with reason string otherwise
+- Use `AbortSignal.timeout(15_000)` for the request
+- Both `cmdSetup` and `cmdSetupInteractive` should call it after saving config
+- `cmdSetup` returns validation result in its return object: `{ ...existing, validation: { ok: true } | { ok: false, error: string } }`
+- `cmdSetupInteractive` prints a warning to console if validation fails, success message if it passes
+- Use the project logger (`createLogger`) — no raw `console.log` except in interactive CLI output (per CLAUDE.md)
+
+## Test Cases (vitest)
+
+### 1. `validateModel` — success path
+- Mock `fetch` to return `{ status: 200, ok: true, json: () => ({}) }`
+- Call `validateModel(baseUrl, apiKey, model)`
+- Assert returns `{ ok: true, value: undefined }`
+- Assert fetch was called with correct URL (`${baseUrl}/chat/completions`), correct headers (`Authorization: Bearer ${apiKey}`), correct body (model, messages, max_tokens: 1)
+
+### 2. `validateModel` — HTTP error (401 unauthorized)
+- Mock `fetch` to return `{ status: 401, ok: false, statusText: "Unauthorized" }`
+- Call `validateModel(baseUrl, apiKey, model)`
+- Assert returns `{ ok: false, error: <string containing "401"> }`
+
+### 3. `validateModel` — HTTP error (404 model not found)
+- Mock `fetch` to return `{ status: 404, ok: false, statusText: "Not Found" }`
+- Assert returns `{ ok: false, error: <string containing "404"> }`
+
+### 4. `validateModel` — network timeout
+- Mock `fetch` to throw `DOMException` with name `AbortError`
+- Assert returns `{ ok: false, error: <string containing "timeout" or "unreachable"> }`
+
+### 5. `validateModel` — network error (DNS failure, connection refused)
+- Mock `fetch` to throw `TypeError("fetch failed")`
+- Assert returns `{ ok: false, error: <string mentioning connectivity> }`
+
+### 6. `cmdSetup` — includes validation result on success
+- Mock global `fetch` for `/chat/completions` to succeed
+- Call `cmdSetup({ provider, baseUrl, apiKey, model, storageRoot })`
+- Assert returned object has `validation: { ok: true, value: undefined }`
+- Assert config files are still written (existing behavior preserved)
+
+### 7. `cmdSetup` — includes validation result on failure (config still saved)
+- Mock global `fetch` for `/chat/completions` to return 401
+- Call `cmdSetup({ ... })`
+- Assert returned object has `validation: { ok: false, error: ... }`
+- Assert `config.yaml` and `.env` are still written (validation failure doesn't prevent saving)
+
+### 8. `cmdSetupInteractive` — prints success message on validation pass
+- Mock `fetch` for both `/models` and `/chat/completions` to succeed
+- Mock stdin to provide valid selections
+- Capture console output
+- Assert output contains a success message like "Model verified" or "✓"
+
+### 9. `cmdSetupInteractive` — prints warning on validation failure
+- Mock `fetch`: `/models` succeeds, `/chat/completions` returns 401
+- Mock stdin for valid selections
+- Capture console output
+- Assert output contains a warning about model not being reachable and suggests trying a different model
+
+### 10. `validateModel` — request body correctness
+- Mock `fetch` to capture the request body
+- Call `validateModel(baseUrl, apiKey, "test-model")`
+- Assert body is `{ model: "test-model", messages: [{role: "user", content: "hi"}], max_tokens: 1 }`
+
+## Export Requirements
+
+- `validateModel` must be exported (for direct unit testing)
+- Signature: `async function validateModel(baseUrl: string, apiKey: string, model: string): Promise<Result<void, string>>`
+- `Result` type: `{ ok: true; value: T } | { ok: false; error: E }` (project convention)
+
+## Files to Create/Modify
+
+- **New**: `packages/cli-workflow/src/__tests__/setup-validate.test.ts` — all test cases above
+- **Modify**: `packages/cli-workflow/src/commands/setup.ts` — add `validateModel`, integrate into `cmdSetup` and `cmdSetupInteractive`

packages/cli-workflow/src/__tests__/setup-validate.test.ts

@@ -0,0 +1,150 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
+import { cmdSetup, validateModel } from "../commands/setup.js";
+
+describe("validateModel", () => {
+  const BASE_URL = "https://api.example.com/v1";
+  const API_KEY = "sk-test-key";
+  const MODEL = "test-model";
+
+  afterEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  test("success path — returns ok on 200", async () => {
+    const mockFetch = vi
+      .spyOn(globalThis, "fetch")
+      .mockResolvedValue(new Response(JSON.stringify({}), { status: 200 }));
+
+    const result = await validateModel(BASE_URL, API_KEY, MODEL);
+
+    expect(result).toEqual({ ok: true, value: undefined });
+    expect(mockFetch).toHaveBeenCalledOnce();
+
+    const [url, opts] = mockFetch.mock.calls[0]!;
+    expect(url).toBe(`${BASE_URL}/chat/completions`);
+    expect((opts as RequestInit).headers).toEqual(
+      expect.objectContaining({ Authorization: `Bearer ${API_KEY}` }),
+    );
+    const body = JSON.parse((opts as RequestInit).body as string);
+    expect(body).toEqual({
+      model: MODEL,
+      messages: [{ role: "user", content: "hi" }],
+      max_tokens: 1,
+    });
+  });
+
+  test("HTTP 401 — returns error containing 401", async () => {
+    vi.spyOn(globalThis, "fetch").mockResolvedValue(
+      new Response("Unauthorized", { status: 401, statusText: "Unauthorized" }),
+    );
+
+    const result = await validateModel(BASE_URL, API_KEY, MODEL);
+
+    expect(result.ok).toBe(false);
+    if (!result.ok) {
+      expect(result.error).toContain("401");
+    }
+  });
+
+  test("HTTP 404 — returns error containing 404", async () => {
+    vi.spyOn(globalThis, "fetch").mockResolvedValue(
+      new Response("Not Found", { status: 404, statusText: "Not Found" }),
+    );
+
+    const result = await validateModel(BASE_URL, API_KEY, MODEL);
+
+    expect(result.ok).toBe(false);
+    if (!result.ok) {
+      expect(result.error).toContain("404");
+    }
+  });
+
+  test("network timeout — returns error mentioning timeout", async () => {
+    const err = new DOMException("signal timed out", "AbortError");
+    vi.spyOn(globalThis, "fetch").mockRejectedValue(err);
+
+    const result = await validateModel(BASE_URL, API_KEY, MODEL);
+
+    expect(result.ok).toBe(false);
+    if (!result.ok) {
+      expect(result.error.toLowerCase()).toMatch(/timeout|timed out/);
+    }
+  });
+
+  test("network error (DNS/connection) — returns error mentioning connectivity", async () => {
+    vi.spyOn(globalThis, "fetch").mockRejectedValue(new TypeError("fetch failed"));
+
+    const result = await validateModel(BASE_URL, API_KEY, MODEL);
+
+    expect(result.ok).toBe(false);
+    if (!result.ok) {
+      expect(result.error.toLowerCase()).toMatch(/connect|reach|network/);
+    }
+  });
+
+  test("request body correctness", async () => {
+    const mockFetch = vi
+      .spyOn(globalThis, "fetch")
+      .mockResolvedValue(new Response(JSON.stringify({}), { status: 200 }));
+
+    await validateModel(BASE_URL, API_KEY, "my-special-model");
+
+    const body = JSON.parse((mockFetch.mock.calls[0]![1] as RequestInit).body as string);
+    expect(body).toEqual({
+      model: "my-special-model",
+      messages: [{ role: "user", content: "hi" }],
+      max_tokens: 1,
+    });
+  });
+});
+
+describe("cmdSetup with validation", () => {
+  let storageRoot: string;
+
+  beforeEach(async () => {
+    storageRoot = await mkdtemp(join(tmpdir(), "uwf-setup-validate-"));
+  });
+
+  afterEach(async () => {
+    vi.restoreAllMocks();
+    await rm(storageRoot, { recursive: true, force: true });
+  });
+
+  const setupArgs = () => ({
+    provider: "testprovider",
+    baseUrl: "https://api.test.com/v1",
+    apiKey: "sk-test",
+    model: "test-model",
+    storageRoot,
+  });
+
+  test("includes validation result on success", async () => {
+    vi.spyOn(globalThis, "fetch").mockResolvedValue(
+      new Response(JSON.stringify({}), { status: 200 }),
+    );
+
+    const result = await cmdSetup(setupArgs());
+
+    expect(result.validation).toEqual({ ok: true, value: undefined });
+    // Config files should still be written
+    expect(result.configPath).toBeTruthy();
+    expect(result.envPath).toBeTruthy();
+  });
+
+  test("includes validation failure — config still saved", async () => {
+    vi.spyOn(globalThis, "fetch").mockResolvedValue(
+      new Response("Unauthorized", { status: 401, statusText: "Unauthorized" }),
+    );
+
+    const result = await cmdSetup(setupArgs());
+
+    expect(result.validation).toBeDefined();
+    expect((result.validation as { ok: boolean }).ok).toBe(false);
+    // Config files should still be written despite validation failure
+    expect(result.configPath).toBeTruthy();
+    expect(result.envPath).toBeTruthy();
+  });
+});

packages/cli-workflow/src/commands/setup.ts

@@ -2,9 +2,45 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
 import { stdin as input, stdout as output } from "node:process";
 import { createInterface } from "node:readline/promises";
-
+import type { Result } from "@uncaged/workflow-util";
 import { parse, stringify } from "yaml";
 
+/**
+ * Send a minimal chat completion request to verify the model is reachable.
+ * Returns ok on 2xx, error with reason string otherwise.
+ */
+export async function validateModel(
+  baseUrl: string,
+  apiKey: string,
+  model: string,
+): Promise<Result<void, string>> {
+  try {
+    const url = `${baseUrl.replace(/\/+$/, "")}/chat/completions`;
+    const res = await fetch(url, {
+      method: "POST",
+      headers: {
+        Authorization: `Bearer ${apiKey}`,
+        "Content-Type": "application/json",
+      },
+      body: JSON.stringify({
+        model,
+        messages: [{ role: "user", content: "hi" }],
+        max_tokens: 1,
+      }),
+      signal: AbortSignal.timeout(15_000),
+    });
+    if (!res.ok) {
+      return { ok: false, error: `HTTP ${res.status} ${res.statusText}` };
+    }
+    return { ok: true, value: undefined };
+  } catch (err: unknown) {
+    if (err instanceof DOMException && err.name === "AbortError") {
+      return { ok: false, error: "Request timed out — model endpoint unreachable" };
+    }
+    return { ok: false, error: `Network error — could not reach endpoint (${String(err)})` };
+  }
+}
+
 /**
  * Preset provider list — embedded to avoid runtime YAML loading dependency.
  * Keep in sync with providers.yaml in cli-workflow.
@@ -163,12 +199,16 @@ export async function cmdSetup(args: SetupArgs): Promise<Record<string, unknown>
   envData[envName] = args.apiKey;
   saveEnvFile(envPath, envData);
 
+  // Validate model connectivity
+  const validation = await validateModel(args.baseUrl, args.apiKey, args.model);
+
   return {
     configPath,
     envPath,
     provider: args.provider,
     model: args.model,
     defaultAgent: merged.defaultAgent,
+    validation,
   };
 }
 
@@ -328,7 +368,7 @@ export async function cmdSetupInteractive(storageRoot: string): Promise<Record<s
 
     console.log(`  → ${providerName}/${model}\n`);
 
-    await cmdSetup({
+    const setupResult = await cmdSetup({
       provider: providerName,
       baseUrl,
       apiKey,
@@ -336,6 +376,19 @@ export async function cmdSetupInteractive(storageRoot: string): Promise<Record<s
       storageRoot,
     });
 
+    // Show validation result
+    if (setupResult.validation && typeof setupResult.validation === "object") {
+      const v = setupResult.validation as { ok: boolean; error?: string };
+      if (v.ok) {
+        console.log("✓ Model verified — connection successful.\n");
+      } else {
+        console.log(`\n⚠ Warning: Could not reach model — ${v.error}`);
+        console.log(
+          "  Config saved, but you may want to try a different model or check your API key.\n",
+        );
+      }
+    }
+
     console.log("Setup complete! Get started:\n");
     console.log("  uwf workflow put <workflow.yaml>   Register a workflow");
     console.log('  uwf thread start <name> -p "..."   Start a thread');

packages/workflow-agent-kit/__tests__/build-output-format-instruction.test.ts

@@ -2,13 +2,32 @@ import { describe, expect, test } from "vitest";
 
 import { buildOutputFormatInstruction } from "../src/build-output-format-instruction.js";
 
+const PLANNER_SCHEMA = {
+  type: "object",
+  properties: {
+    status: { type: "string", enum: ["ready", "insufficient_info"] },
+    plan: { type: "string" },
+  },
+  required: ["status"],
+  additionalProperties: false,
+};
+
+const REVIEWER_SCHEMA = {
+  type: "object",
+  properties: {
+    approved: { type: "boolean" },
+  },
+  required: ["approved"],
+  additionalProperties: false,
+};
+
 describe("buildOutputFormatInstruction", () => {
   test("always includes the frontmatter example block", () => {
     const result = buildOutputFormatInstruction({});
     expect(result).toContain("---");
-    expect(result).toContain("status: done");
-    expect(result).toContain("confidence:");
-    expect(result).toContain("scope: role");
+    expect(result).not.toContain("status: done");
+    expect(result).not.toContain("confidence:");
+    expect(result).not.toContain("scope: role");
   });
 
   test("always marks frontmatter as the primary deliverable", () => {
@@ -16,17 +35,36 @@ describe("buildOutputFormatInstruction", () => {
     expect(result).toContain("primary deliverable");
   });
 
-  test("lists fields from a flat object schema", () => {
+  test("generates planner-specific YAML example from schema", () => {
+    const result = buildOutputFormatInstruction(PLANNER_SCHEMA);
+    expect(result).toContain("status: ready  # required | ready | insufficient_info");
+    expect(result).toContain("plan: <string>");
+    expect(result).not.toContain("status: done");
+    expect(result).not.toContain("confidence:");
+    expect(result).not.toContain("artifacts:");
+  });
+
+  test("generates reviewer-specific YAML example from schema", () => {
+    const result = buildOutputFormatInstruction(REVIEWER_SCHEMA);
+    expect(result).toContain("approved: true  # required | true | false");
+    expect(result).not.toContain("status:");
+  });
+
+  test("lists fields from a flat object schema with required marker", () => {
     const schema = {
       type: "object",
       properties: {
         status: { type: "string" },
         confidence: { type: "number" },
       },
+      required: ["status"],
     };
     const result = buildOutputFormatInstruction(schema);
-    expect(result).toContain("`status`");
+    expect(result).toContain("`status` (required)");
     expect(result).toContain("`confidence`");
+    expect(result).not.toContain("`confidence` (required)");
+    expect(result).toContain("status: <string>  # required");
+    expect(result).toContain("confidence: <number>");
   });
 
   test("lists union of fields from an anyOf schema", () => {
@@ -45,6 +83,8 @@ describe("buildOutputFormatInstruction", () => {
     const result = buildOutputFormatInstruction(schema);
     expect(result).toContain("`alpha`");
     expect(result).toContain("`beta`");
+    expect(result).toContain("alpha: <string>");
+    expect(result).toContain("beta: <number>");
   });
 
   test("lists union of fields from a oneOf schema", () => {
@@ -63,6 +103,8 @@ describe("buildOutputFormatInstruction", () => {
     const result = buildOutputFormatInstruction(schema);
     expect(result).toContain("`foo`");
     expect(result).toContain("`bar`");
+    expect(result).toContain("foo: <string>");
+    expect(result).toContain("bar: true  # true | false");
   });
 
   test("falls back gracefully for a non-object schema with no properties", () => {
@@ -80,6 +122,23 @@ describe("buildOutputFormatInstruction", () => {
     const result = buildOutputFormatInstruction(schema);
     const matches = [...result.matchAll(/`shared`/g)];
     expect(matches.length).toBe(1);
+    expect(result).toContain("shared: <string>");
+  });
+
+  test("marks required when any union variant requires the field", () => {
+    const schema = {
+      anyOf: [
+        {
+          type: "object",
+          properties: { shared: { type: "string" } },
+          required: ["shared"],
+        },
+        { type: "object", properties: { shared: { type: "number" } } },
+      ],
+    };
+    const result = buildOutputFormatInstruction(schema);
+    expect(result).toContain("`shared` (required)");
+    expect(result).toContain("shared: <string>  # required");
   });
 
   test("includes focus reminder about role scope", () => {

packages/workflow-agent-kit/src/build-output-format-instruction.ts

@@ -1,5 +1,11 @@
 import type { JSONSchema } from "@uncaged/json-cas";
 
+type SchemaProperty = {
+  name: string;
+  schema: JSONSchema;
+  required: boolean;
+};
+
 /**
  * Extract top-level property names from a JSON Schema object.
  *
@@ -10,8 +16,43 @@ import type { JSONSchema } from "@uncaged/json-cas";
  * Returns an empty array for schemas with no inspectable property definitions.
  */
 export function extractSchemaFields(schema: JSONSchema): string[] {
+  return extractSchemaProperties(schema).map((p) => p.name);
+}
+
+function extractSchemaProperties(schema: JSONSchema): SchemaProperty[] {
+  const objectSchemas = collectObjectSchemas(schema);
+  if (objectSchemas.length === 0) {
+    return [];
+  }
+
+  const byName = new Map<string, SchemaProperty>();
+
+  for (const objectSchema of objectSchemas) {
+    const requiredSet = new Set(
+      Array.isArray(objectSchema.required) ? (objectSchema.required as string[]) : [],
+    );
+    const properties = objectSchema.properties as Record<string, JSONSchema> | null | undefined;
+    if (typeof properties !== "object" || properties === null) {
+      continue;
+    }
+
+    for (const [name, propSchema] of Object.entries(properties)) {
+      const required = requiredSet.has(name);
+      const existing = byName.get(name);
+      if (existing === undefined) {
+        byName.set(name, { name, schema: propSchema, required });
+      } else if (required) {
+        byName.set(name, { ...existing, required: true });
+      }
+    }
+  }
+
+  return [...byName.values()];
+}
+
+function collectObjectSchemas(schema: JSONSchema): JSONSchema[] {
   if (typeof schema.properties === "object" && schema.properties !== null) {
-    return Object.keys(schema.properties as Record<string, unknown>);
+    return [schema];
   }
 
   const unionKey = Array.isArray(schema.anyOf)
@@ -20,18 +61,109 @@ export function extractSchemaFields(schema: JSONSchema): string[] {
       ? "oneOf"
       : null;
 
-  if (unionKey !== null) {
-    const variants = schema[unionKey] as JSONSchema[];
-    const fieldSet = new Set<string>();
-    for (const variant of variants) {
-      for (const field of extractSchemaFields(variant)) {
-        fieldSet.add(field);
-      }
-    }
-    return [...fieldSet];
+  if (unionKey === null) {
+    return [];
   }
 
-  return [];
+  const variants = schema[unionKey] as JSONSchema[];
+  const result: JSONSchema[] = [];
+  for (const variant of variants) {
+    result.push(...collectObjectSchemas(variant));
+  }
+  return result;
+}
+
+function resolvePropertySchema(prop: JSONSchema): JSONSchema {
+  if (Array.isArray(prop.enum) && prop.enum.length > 0) {
+    return prop;
+  }
+
+  const unionKey = Array.isArray(prop.anyOf) ? "anyOf" : Array.isArray(prop.oneOf) ? "oneOf" : null;
+
+  if (unionKey !== null) {
+    const variants = prop[unionKey] as JSONSchema[];
+    const nonNull = variants.filter((v) => v.type !== "null");
+    if (nonNull.length === 1) {
+      return nonNull[0];
+    }
+  }
+
+  return prop;
+}
+
+function formatYamlScalar(value: unknown): string {
+  if (typeof value === "boolean") {
+    return String(value);
+  }
+  if (typeof value === "number") {
+    return String(value);
+  }
+  return String(value);
+}
+
+function buildPropertyComment(parts: string[]): string {
+  const filtered = parts.filter((p) => p.length > 0);
+  return filtered.length > 0 ? `  # ${filtered.join(" | ")}` : "";
+}
+
+function buildPropertyExampleLine(prop: SchemaProperty): string {
+  const resolved = resolvePropertySchema(prop.schema);
+  const commentParts: string[] = [];
+  if (prop.required) {
+    commentParts.push("required");
+  }
+
+  if (Array.isArray(resolved.enum) && resolved.enum.length > 0) {
+    const enumValues = resolved.enum.map((v) => String(v));
+    commentParts.push(...enumValues);
+    const first = resolved.enum[0];
+    return `${prop.name}: ${formatYamlScalar(first)}${buildPropertyComment(commentParts)}`;
+  }
+
+  if (resolved.type === "boolean") {
+    commentParts.push("true", "false");
+    return `${prop.name}: true${buildPropertyComment(commentParts)}`;
+  }
+
+  if (resolved.type === "string") {
+    return `${prop.name}: <string>${buildPropertyComment(commentParts)}`;
+  }
+
+  if (resolved.type === "number" || resolved.type === "integer") {
+    return `${prop.name}: <number>${buildPropertyComment(commentParts)}`;
+  }
+
+  if (resolved.type === "array") {
+    return `${prop.name}:\n  - <item>${buildPropertyComment(commentParts)}`;
+  }
+
+  if (resolved.type === "object") {
+    return `${prop.name}: <object>${buildPropertyComment(commentParts)}`;
+  }
+
+  return `${prop.name}: <value>${buildPropertyComment(commentParts)}`;
+}
+
+function buildYamlExampleBlock(properties: SchemaProperty[]): string {
+  if (properties.length === 0) {
+    return "---\n\n... your markdown work here ...";
+  }
+
+  const lines = properties.map((p) => buildPropertyExampleLine(p));
+  return `---\n${lines.join("\n")}\n---\n\n... your markdown work here ...`;
+}
+
+function buildFieldList(properties: SchemaProperty[]): string {
+  if (properties.length === 0) {
+    return "  (schema fields will be extracted automatically)";
+  }
+
+  return properties
+    .map((p) => {
+      const suffix = p.required ? " (required)" : "";
+      return `  - \`${p.name}\`${suffix}`;
+    })
+    .join("\n");
 }
 
 /**
@@ -42,28 +174,16 @@ export function extractSchemaFields(schema: JSONSchema): string[] {
  * system prompt so the deliverable format is the first thing the agent sees.
  */
 export function buildOutputFormatInstruction(schema: JSONSchema): string {
-  const fields = extractSchemaFields(schema);
-
-  const fieldList =
-    fields.length > 0
-      ? fields.map((f) => `  - \`${f}\``).join("\n")
-      : "  (schema fields will be extracted automatically)";
+  const properties = extractSchemaProperties(schema);
+  const yamlExample = buildYamlExampleBlock(properties);
+  const fieldList = buildFieldList(properties);
 
   return `## Deliverable Format
 
 Your response MUST begin with a YAML frontmatter block followed by your markdown work:
 
 \`\`\`
----
-status: done          # done | needs_input | in_progress | failed
-next: <role-name>     # suggested next role, or omit
-confidence: 0.9       # 0.0–1.0, your self-assessed confidence
-artifacts:            # list of file paths or CAS hashes you produced
-  - path/to/file.ts
-scope: role           # role | thread
----
-
-... your markdown work here ...
+${yamlExample}
 \`\`\`
 
 The frontmatter is the **primary deliverable** — the engine reads it directly.