From a7e6caf6e783fa07847debdf23c7c96d9073c5a3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=B0=8F=E6=A9=98?= Date: Fri, 24 Apr 2026 21:47:37 +0000 Subject: [PATCH] docs: update all README files to match actual code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrite documentation across all packages to reflect current architecture, APIs, and CLI commands. - README.md: fix reflex examples, add store package, update config - core/README.md: add Sense→workflow routing, IPC types - daemon/README.md: complete module table, crash recovery, createKernel - cli/README.md: add workflow/sense/store subcommands - store/README.md: new file documenting LogStore/BlobStore Fixes #95 --- README.md | 112 +++++++++++++++++++++++--------------- packages/cli/README.md | 46 +++++++++++----- packages/core/README.md | 32 ++++++++++- packages/daemon/README.md | 71 ++++++++++++++++++------ packages/store/README.md | 48 ++++++++++++++++ 5 files changed, 230 insertions(+), 79 deletions(-) create mode 100644 packages/store/README.md diff --git a/README.md b/README.md index 66c88dd..a75cfcd 100644 --- a/README.md +++ b/README.md @@ -7,28 +7,31 @@ Nerve is a lightweight daemon that continuously observes external state through ## Core Concepts ``` -External World → Sense → Signal → Reflex → Workflow → Log - ↑ ↑ - "what to observe" "what to do" +External World → Sense ─┬→ Signal → Reflex → Sense (scheduled compute) + │ + └→ Workflow (Sense return with workflow directive) → Log ``` | Concept | Metaphor | Role | |---------|----------|------| | **Sense** | 👁️ Perception | A `compute()` function that samples or derives data. Each sense has its own SQLite database. | -| **Reflex** | ⚡ Reaction | Declarative trigger — interval-based, event-driven, or both. Connects senses to actions. | -| **Signal** | 📡 Notification | Emitted when a sense returns non-null. Other reflexes can listen for signals. | -| **Workflow** | 🔧 Action | Stateful multi-step execution with Roles (actors) and a Moderator (coordinator). | +| **Reflex** | ⚡ Reaction | Declarative rules that **only schedule Sense computes** (interval and/or `on` signal names). Reflex YAML cannot reference workflows. | +| **Signal** | 📡 Notification | Emitted when a sense returns a non-null value that is routed as a normal signal (see Sense → Workflow below). Other reflexes can listen via `on`. | +| **Workflow** | 🔧 Action | Stateful multi-step execution with Roles and a Moderator. Started from a Sense return value or from CLI/daemon IPC—not from reflex YAML. | | **Log** | 📝 Record | Immutable audit trail. Queryable by senses, but **cannot** trigger reflexes (prevents feedback loops). | -Three extension points, fully orthogonal — a Sense doesn't know when it runs, a Reflex doesn't know what it computes, a Workflow doesn't know why it was triggered. +**Sense → Workflow:** if `compute()` returns a plain object with a string field `workflow` in the form `name|maxRounds|prompt` (only the first two `|` delimit name and rounds; the rest is the prompt), the engine starts that workflow and **does not** emit a Signal for that return. `workflow: null` or `""` means “emit a signal” and strip the key from the payload. Invalid `workflow` strings are treated like a normal signal (directive stripped). See `@uncaged/nerve-core` `routeSenseComputeOutput` / `parseSenseWorkflowDirective`. + +Three extension points for **what / when / multi-step action** — reflexes never replace Sense-driven workflow launches. ## Packages | Package | Description | |---------|-------------| -| [`@uncaged/nerve-core`](./packages/core) | Shared types and config parser | -| [`@uncaged/nerve-daemon`](./packages/daemon) | The observation engine — kernel, sense runtime, reflex scheduler, workflow manager | -| [`@uncaged/nerve-cli`](./packages/cli) | CLI tool (`nerve`) — init, start, stop, logs, query | +| [`@uncaged/nerve-core`](./packages/core) | Shared types, config parser, Sense→workflow routing, daemon IPC protocol | +| [`@uncaged/nerve-store`](./packages/store) | Append-only log SQLite, JSONL archive, CAS blob store, workflow run rows | +| [`@uncaged/nerve-daemon`](./packages/daemon) | Kernel, workers, signal bus, reflex scheduler, workflow manager, file watcher, IPC | +| [`@uncaged/nerve-cli`](./packages/cli) | CLI (`nerve`) — init, validate, daemon, dev, logs, sense, store, workflow | ## Quick Start @@ -70,15 +73,17 @@ nerve logs # view logs ## Configuration -`nerve.yaml` declares senses, reflexes, and workflows: +`nerve.yaml` declares senses, reflexes (sense-only), optional workflows (concurrency), and optional engine `max_rounds`: ```yaml +max_rounds: 100 # default moderator cap (e.g. CLI workflow trigger) + senses: cpu-usage: group: system # senses in the same group share a worker process throttle: 10s # min interval between computes timeout: 30s # max compute duration - gracePeriod: 5s # wait before first compute after startup + grace_period: 5s # wait before first compute after startup reflexes: - kind: sense @@ -86,10 +91,6 @@ reflexes: interval: 30s # periodic trigger on: [disk-pressure] # also trigger on signals from other senses - - kind: workflow - workflow: cleanup - on: [disk-pressure] # start a workflow when signal fires - workflows: cleanup: concurrency: 1 @@ -97,43 +98,66 @@ workflows: code-review: concurrency: 3 overflow: queue - maxQueue: 20 + max_queue: 20 +``` + +YAML must **not** include `workflow:` under `reflexes` — the parser rejects it. Declare workflows under `workflows:` and start them from Sense `compute()` or `nerve workflow trigger`. + +**Example — Sense starts a workflow** (`senses/disk-pressure/compute.ts`): + +```typescript +export async function compute() { + const full = await diskNearlyFull(); + if (!full) return null; + return { + path: "/data", + workflow: "cleanup|10|Disk partition nearly full", // name|maxRounds|prompt + }; +} ``` ## Architecture ``` -┌─────────────────────────────────────────────────────────┐ -│ Kernel │ -│ │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │ Worker │ │ Worker │ │ Worker │ (1 per │ -│ │ (group A)│ │ (group B)│ │ (group C)│ group) │ -│ │ sense-1 │ │ sense-3 │ │ sense-5 │ │ -│ │ sense-2 │ │ sense-4 │ │ │ │ -│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ -│ │ │ │ │ -│ └──────────────┼──────────────┘ │ -│ ▼ │ -│ ┌──────────────┐ │ -│ │ Signal Bus │ │ -│ └──────┬───────┘ │ -│ ▼ │ -│ ┌──────────────────┐ │ -│ │ Reflex Scheduler │ │ -│ └────────┬─────────┘ │ -│ ▼ │ -│ ┌───────────────────┐ │ -│ │ Workflow Manager │──→ Log Store (SQLite) │ -│ └───────────────────┘ │ -└─────────────────────────────────────────────────────────┘ +┌────────────────────────────────────────────────────────────────────────┐ +│ Kernel │ +│ │ +│ ┌──────────────┐ watches nerve.yaml / senses / workflows │ +│ │ File Watcher ├──────────────────────────────────────────┐ │ +│ └──────────────┘ │ │ +│ ┌──────────────┐ CLI ↔ newline JSON (trigger-workflow, │ │ +│ │ Daemon IPC │ trigger-sense, list-senses) │ │ +│ └──────┬───────┘ ▼ │ +│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ │ Worker │ │ Worker │ │ Worker │ (1 per│ +│ │ │ (group A)│ │ (group B)│ │ (group C)│ group) │ +│ │ │ sense-1 │ │ sense-3 │ │ sense-5 │ │ +│ │ │ sense-2 │ │ sense-4 │ │ │ │ +│ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ +│ │ │ │ │ │ +│ │ └──────────────┼──────────────┘ │ +│ │ ▼ │ +│ │ ┌──────────────┐ │ +│ │ │ Signal Bus │ │ +│ │ └──────┬───────┘ │ +│ │ ▼ │ +│ │ ┌──────────────────┐ │ +│ │ │ Reflex Scheduler│ │ +│ │ └────────┬─────────┘ │ +│ │ ▼ │ +│ │ ┌───────────────────┐ │ +│ └───────────────────►│ Workflow Manager │──→ @uncaged/nerve-store │ +│ └───────────────────┘ (logs.db, …) │ +└────────────────────────────────────────────────────────────────────────┘ ``` -- **Worker processes** — one per sense group, forked by the kernel. Isolated compute execution. +- **Worker pool** — one child process per sense group; isolation between groups. - **Signal Bus** — in-memory pub/sub for signal distribution. - **Reflex Scheduler** — interval timers + signal subscriptions, with throttle/coalesce. -- **Workflow Manager** — concurrency control (drop/queue), thread lifecycle tracking. -- **Log Store** — WAL-mode SQLite via `node:sqlite`, with archival and retention policies. +- **Workflow Manager** — concurrency (drop/queue), per-workflow workers, crash recovery. +- **File watcher** — hot reload for config, sense modules, and workflow modules. +- **Daemon IPC** — Unix domain socket; used by the CLI when the daemon is running. +- **Log / blob storage** — implemented in `@uncaged/nerve-store` (WAL SQLite, JSONL archive, CAS blobs). ## Tech Stack diff --git a/packages/cli/README.md b/packages/cli/README.md index 607a592..8c548ee 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -21,41 +21,59 @@ nerve init # Initialize a nerve workspace (installs deps, scaff nerve validate # Validate nerve.yaml configuration ``` -### Daemon Management +### Daemon management ```bash nerve daemon start # Start the daemon (background) nerve daemon stop # Stop the daemon -nerve daemon status # Check daemon health -nerve daemon restart # Restart the daemon -nerve daemon logs # Tail daemon logs +nerve daemon status # Show pid, uptime, sense names from nerve.yaml (process must exist) +nerve daemon restart # Stop then start +nerve daemon logs # Tail daemon process logs (file under workspace logs/) ``` ### Development ```bash -nerve dev # Run in foreground mode (no daemon, Ctrl+C to stop) +nerve dev # Foreground kernel with file watcher + IPC (Ctrl+C stops) ``` -### Querying +### Querying & status ```bash -nerve logs # View structured logs -nerve sense query # Query a sense's SQLite database -nerve sense schema # Show a sense's database schema -nerve status # Daemon health summary +nerve logs # Tail or page the daemon text log file (path in footer; default ~/.uncaged-nerve/logs/nerve.log) +nerve status # Short daemon health summary (aliases daemon status) +``` + +Structured rows in `data/logs.db` are surfaced via **`nerve workflow inspect`** / **`nerve workflow list`** (and `LogStore` in code), not via `nerve logs`. + +### Sense + +```bash +nerve sense list # List senses (live fields from daemon IPC when running) +nerve sense trigger # IPC trigger-sense — queue a compute for that sense +nerve sense query # Read-only SQL on data/senses/.db (optional SQL args) +nerve sense schema # Print CREATE TABLE statements for that sense DB +``` + +### Store maintenance + +```bash +nerve store archive # Move old log rows to JSONL under data/archive/logs/… (optional --vacuum) ``` ### Workflows ```bash -nerve workflow list # List workflow runs -nerve workflow show # Show workflow run details +nerve workflow list # Queued/started runs (add --all for terminal states; --workflow, --limit, --offset) +nerve workflow inspect # Run metadata + paginated workflow log lines +nerve workflow thread # Role rounds from persisted messages (--before, --budget) +nerve workflow trigger # IPC trigger-workflow (daemon must be running) + # Optional JSON: --payload '{"prompt":"…","maxRounds":50}' ``` -### Top-level Aliases +`nerve workflow trigger` sends a `trigger-workflow` line on the daemon Unix socket (same protocol as `@uncaged/nerve-core` / `parseDaemonIpcRequest`). It does not read `nerve.yaml` workflow definitions beyond what the running daemon already loaded. -For convenience, these aliases are available: +### Top-level aliases ```bash nerve start → nerve daemon start diff --git a/packages/core/README.md b/packages/core/README.md index 17738db..8e53e3b 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -4,9 +4,12 @@ Shared types and configuration parser for the [nerve](../../README.md) observati ## What's Inside -- **Type definitions** — `Signal`, `SenseConfig`, `ReflexConfig`, `WorkflowConfig`, `NerveConfig`, and all related types -- **Config parser** — `parseNerveConfig(yaml)` validates and parses `nerve.yaml` into a typed `NerveConfig` -- **Result type** — `Result` with `ok()` / `err()` helpers for explicit error handling (no thrown exceptions) +- **Type definitions** — `Signal`, `SenseConfig`, `SenseInfo`, `SenseReflexConfig`, `ReflexConfig` (sense-only), `WorkflowConfig`, `NerveConfig`, and related types +- **Config parser** — `parseNerveConfig(yaml)` validates and parses `nerve.yaml` into `NerveConfig` (rejects reflex entries that declare a `workflow` key; reflexes only schedule senses) +- **Sense → workflow routing** — `parseSenseWorkflowDirective`, `routeSenseComputeOutput`, and types `ParsedSenseWorkflowDirective`, `SenseComputeRoute` +- **Daemon IPC protocol** — request/response types (`DaemonIpcRequest`, `DaemonIpcResponse`, …) and `parseDaemonIpcRequest` for newline-delimited JSON on the CLI ↔ daemon socket +- **Workflow automaton types** — `START` / `END` sentinel constants, `WorkflowMessage`, `StartSignal`, `RoleSignal`, `Moderator`, `WorkflowDefinition`, `Role`, `SenseResult`, plus `DEFAULT_ENGINE_MAX_ROUNDS` +- **Result type** — `Result` with `ok()` / `err()` helpers for explicit error handling (no thrown exceptions for parse paths) ## Usage @@ -20,6 +23,29 @@ if (result.ok) { } ``` +### Sense return → signal vs workflow + +```typescript +import { parseSenseWorkflowDirective, routeSenseComputeOutput } from "@uncaged/nerve-core"; + +const directive = parseSenseWorkflowDirective("my-workflow|8|Hello from sense"); +if (directive.ok) { + console.log(directive.value.workflowName, directive.value.maxRounds, directive.value.prompt); +} + +const route = routeSenseComputeOutput({ + metric: 42, + workflow: "my-workflow|8|Run now", +}); +if (route.kind === "launch") { + // engine starts workflow; no Signal to the bus for this return + console.log(route.launch); +} else { + // normal signal with payload + console.log(route.payload); +} +``` + ## Duration Format Config fields like `throttle`, `timeout`, and `interval` accept human-readable durations: diff --git a/packages/daemon/README.md b/packages/daemon/README.md index 430703f..997761c 100644 --- a/packages/daemon/README.md +++ b/packages/daemon/README.md @@ -4,18 +4,33 @@ The observation engine runtime for [nerve](../../README.md) — runs senses, rou ## Architecture -| Module | Responsibility | -|--------|---------------| -| **Kernel** | Top-level orchestrator — spawns workers, wires up signal bus, scheduler, and workflow manager. Supports hot reload and graceful shutdown. | -| **Sense Runtime** | Per-sense SQLite database (via `node:sqlite` + Drizzle ORM), migration runner, peer DB read access. | -| **Sense Worker** | Forked child process — one per sense group. Runs compute functions in isolation. | -| **Signal Bus** | In-memory pub/sub. Sense computes emit signals; reflexes and workflows subscribe. | -| **Reflex Scheduler** | Drives compute triggers — interval timers, signal-based events, throttle/coalesce logic. | -| **Workflow Manager** | Concurrency control (drop/queue), thread lifecycle, worker process management (RFC-002). | -| **Log Store** | Structured log storage in WAL-mode SQLite. Supports retention policies, archival to JSONL, and workflow run tracking. | -| **Blob Store** | Binary artifact storage for workflow outputs. | -| **File Watcher** | Watches `nerve.yaml` and sense files for hot reload. | -| **Daemon IPC** | Unix socket server for CLI ↔ daemon communication. | +| Module | Source (indicative) | Responsibility | +|--------|---------------------|----------------| +| **Kernel** | `kernel.ts` | Orchestrator — worker pool, signal bus, reflex scheduler, workflow manager, optional file watcher and daemon IPC, config reload hooks | +| **Worker pool** | `worker-pool.ts` | Fork and supervise one child process per sense group; restart/shutdown; crash cleanup hooks for scheduler state | +| **Kernel sense groups** | `kernel-sense-groups.ts` | Derive sense groups from config; list senses per group for scheduling | +| **Sense runtime** | sense worker + Drizzle | Per-sense SQLite (`node:sqlite`), migrations, peer DB reads | +| **Sense worker** | `sense-worker.ts` (fork target) | Child process entry — runs `compute()` per sense in a group | +| **Signal bus** | `signal-bus.ts` | In-memory pub/sub for sense signals | +| **Reflex scheduler** | `reflex-scheduler.ts` | Interval + `on` subscriptions, throttle/coalesce | +| **Workflow manager** | `workflow-manager.ts` | One worker per workflow name, concurrency (drop/queue), queue caps | +| **Workflow worker** | `workflow-worker.ts` | Child process — runs RFC-002 threads (`start-thread`, `resume-thread` IPC) | +| **IPC (parent ↔ workers)** | `ipc.ts` | Typed messages for sense and workflow workers (includes `resume-thread` for recovery) | +| **Log / workflow persistence** | via `@uncaged/nerve-store` | Structured logs, `workflow_runs`, thread messages (used for recovery) | +| **Blob store** | `@uncaged/nerve-store` | CAS under `data/blobs/` — sense workers construct `createBlobStore(join(nerveRoot, "data", "blobs"))` for artifact writes | +| **File watcher** | `file-watcher.ts` | Watches workspace paths for config / sense / workflow file changes | +| **Kernel file watch** | `kernel-file-watch.ts` | Maps watcher events to `reloadConfig`, sense group restart, workflow `drainAndRespawn` | +| **Daemon IPC** | `daemon-ipc.ts` | Unix socket server — parses `@uncaged/nerve-core` `DaemonIpcRequest`, dispatches trigger-workflow / trigger-sense / list-senses | + +## Crash recovery (workflow workers) + +If a workflow worker exits unexpectedly while threads are active: + +- In-flight runs are marked **`crashed`** in the log store; the manager respawns a fresh worker. +- Runs still in **`started`** state can be **`resume-thread`**’d: the manager rebuilds the message chain from persisted workflow log rows and sends `resume-thread` to the new worker. +- **Crash-loop backoff:** repeated crashes for the same workflow name are counted in a sliding window (`60s`); after **`5`** crashes in that window, the manager **stops respawning** that worker and logs the condition (avoids tight crash loops). + +Hot reload (`drainAndRespawn`) uses a controlled drain: in-flight runs may be marked **`interrupted`** when the old worker is torn down after a timeout — that path is distinct from unexpected crash recovery. ## Key Design Decisions @@ -26,24 +41,44 @@ The observation engine runtime for [nerve](../../README.md) — runs senses, rou ## Usage -The daemon is typically started via the CLI (`nerve daemon start`), but can be used programmatically: +The daemon is typically started via the CLI (`nerve daemon start` / `nerve dev`), but you can embed the kernel: ```typescript +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { parseNerveConfig } from "@uncaged/nerve-core"; import { createKernel } from "@uncaged/nerve-daemon"; -const kernel = await createKernel(nerveRoot); +const nerveRoot = "/path/to/workspace"; +const yamlPath = join(nerveRoot, "nerve.yaml"); +const parsed = parseNerveConfig(readFileSync(yamlPath, "utf8")); +if (!parsed.ok) { + throw parsed.error; +} + +const kernel = createKernel(parsed.value, nerveRoot, { + enableFileWatcher: true, + ipcSocketPath: join(nerveRoot, "nerve.sock"), +}); + await kernel.ready; -// Trigger a sense manually kernel.triggerSense("cpu-usage"); - -// Check health const health = kernel.getHealth(); -// Graceful shutdown await kernel.stop(); ``` +`createKernel(config, nerveRoot, options?)` — `config` is a parsed `NerveConfig`; `nerveRoot` is the workspace root (contains `nerve.yaml`, `data/`, etc.). Optional `KernelOptions`: + +| Field | Meaning | +|-------|---------| +| `workerScript` | Override path to the sense worker entry script (defaults to the package’s resolved worker) | +| `enableFileWatcher` | Watch config / senses / workflows for hot reload | +| `logStore` | Inject a `LogStore` instance (defaults to `createLogStore(join(nerveRoot, "data", "logs.db"))`) | +| `ipcSocketPath` | When non-null, listen for daemon IPC on this Unix socket path | + ## Install ```bash diff --git a/packages/store/README.md b/packages/store/README.md new file mode 100644 index 0000000..e8835f9 --- /dev/null +++ b/packages/store/README.md @@ -0,0 +1,48 @@ +# @uncaged/nerve-store + +Persistent storage for the [nerve](../../README.md) daemon — append-only structured logs, optional JSONL cold archive, and content-addressable blobs. + +## LogStore (`createLogStore`, `log-store.ts`) + +- **Append-only log table** — rows with `source`, `type`, `refId`, `payload`, `ts` (string payloads for ad hoc fields) +- **SQLite WAL** — `DatabaseSync` from `node:sqlite` +- **Workflow run tracking** — materialized `workflow_runs` table plus helpers to list active runs, upsert status transitions, and read **thread messages** / **role rounds** for CLI and crash recovery +- **Meta key-value** — small `meta` table (e.g. archive watermarks) + +Public exports include `LogStore`, `LogEntry`, `LogQuery`, `WorkflowRun`, `WorkflowRunStatus`, `ThreadRoundRow`, `GetThreadRoundsParams`, and archive-related types re-exported from `log-archive`. + +## WorkflowRunStatus + +Runs progress through a small state machine. Typical paths: + +1. **`queued`** → **`started`** when a worker picks up the thread +2. **`started`** → **`completed`** | **`failed`** | **`crashed`** | **`interrupted`** | **`dropped`** + +Semantics in the daemon/store layer: + +- **`completed` / `failed`** — normal terminal outcomes from the workflow worker +- **`crashed`** — worker exited unexpectedly; manager may respawn and **`resume-thread`** eligible **`started`** runs +- **`interrupted`** — e.g. hot-reload drain killed an in-flight thread after timeout +- **`dropped`** — concurrency **`overflow: drop`** rejected a new run, or **`overflow: queue`** evicted an queued item when the queue was full + +## LogArchive (`log-archive.ts`) + +- **`archiveLogs`** / helpers — export eligible UTC days of old rows to **`data/archive/logs/YYYY-MM-DD.jsonl`**, delete archived rows from SQLite, optional **`VACUUM`** +- Used by **`nerve store archive`** in `@uncaged/nerve-cli` + +## BlobStore (`createBlobStore`, `blob-store.ts`) + +- **Content-addressable storage** — `write` returns lowercase **sha256** hex; files live under **`data/blobs/<2-hex>/<62-hex>`** +- **`read` / `exists`** — path must match digest on disk (tamper detection) + +## Install + +```bash +pnpm add @uncaged/nerve-store +``` + +Requires Node.js ≥ 22.5 (same as the rest of the stack). + +## License + +MIT -- 2.43.0