docs: update all README files to match actual code #96

Merged
xiaomo merged 1 commits from docs/95-update-readme-to-match-code into main 2026-04-24 21:49:34 +00:00
5 changed files with 230 additions and 79 deletions
+68 -44
View File
@@ -7,28 +7,31 @@ Nerve is a lightweight daemon that continuously observes external state through
## Core Concepts
```
External World → Sense → Signal → Reflex → Workflow → Log
↑ ↑
"what to observe" "what to do"
External World → Sense ─┬→ Signal → Reflex → Sense (scheduled compute)
└→ Workflow (Sense return with workflow directive) → Log
```
| Concept | Metaphor | Role |
|---------|----------|------|
| **Sense** | 👁️ Perception | A `compute()` function that samples or derives data. Each sense has its own SQLite database. |
| **Reflex** | ⚡ Reaction | Declarative trigger — interval-based, event-driven, or both. Connects senses to actions. |
| **Signal** | 📡 Notification | Emitted when a sense returns non-null. Other reflexes can listen for signals. |
| **Workflow** | 🔧 Action | Stateful multi-step execution with Roles (actors) and a Moderator (coordinator). |
| **Reflex** | ⚡ Reaction | Declarative rules that **only schedule Sense computes** (interval and/or `on` signal names). Reflex YAML cannot reference workflows. |
| **Signal** | 📡 Notification | Emitted when a sense returns a non-null value that is routed as a normal signal (see Sense → Workflow below). Other reflexes can listen via `on`. |
| **Workflow** | 🔧 Action | Stateful multi-step execution with Roles and a Moderator. Started from a Sense return value or from CLI/daemon IPC—not from reflex YAML. |
| **Log** | 📝 Record | Immutable audit trail. Queryable by senses, but **cannot** trigger reflexes (prevents feedback loops). |
Three extension points, fully orthogonal — a Sense doesn't know when it runs, a Reflex doesn't know what it computes, a Workflow doesn't know why it was triggered.
**Sense → Workflow:** if `compute()` returns a plain object with a string field `workflow` in the form `name|maxRounds|prompt` (only the first two `|` delimit name and rounds; the rest is the prompt), the engine starts that workflow and **does not** emit a Signal for that return. `workflow: null` or `""` means “emit a signal” and strip the key from the payload. Invalid `workflow` strings are treated like a normal signal (directive stripped). See `@uncaged/nerve-core` `routeSenseComputeOutput` / `parseSenseWorkflowDirective`.
Three extension points for **what / when / multi-step action** — reflexes never replace Sense-driven workflow launches.
## Packages
| Package | Description |
|---------|-------------|
| [`@uncaged/nerve-core`](./packages/core) | Shared types and config parser |
| [`@uncaged/nerve-daemon`](./packages/daemon) | The observation engine — kernel, sense runtime, reflex scheduler, workflow manager |
| [`@uncaged/nerve-cli`](./packages/cli) | CLI tool (`nerve`) — init, start, stop, logs, query |
| [`@uncaged/nerve-core`](./packages/core) | Shared types, config parser, Sense→workflow routing, daemon IPC protocol |
| [`@uncaged/nerve-store`](./packages/store) | Append-only log SQLite, JSONL archive, CAS blob store, workflow run rows |
| [`@uncaged/nerve-daemon`](./packages/daemon) | Kernel, workers, signal bus, reflex scheduler, workflow manager, file watcher, IPC |
| [`@uncaged/nerve-cli`](./packages/cli) | CLI (`nerve`) — init, validate, daemon, dev, logs, sense, store, workflow |
## Quick Start
@@ -70,15 +73,17 @@ nerve logs # view logs
## Configuration
`nerve.yaml` declares senses, reflexes, and workflows:
`nerve.yaml` declares senses, reflexes (sense-only), optional workflows (concurrency), and optional engine `max_rounds`:
```yaml
max_rounds: 100 # default moderator cap (e.g. CLI workflow trigger)
senses:
cpu-usage:
group: system # senses in the same group share a worker process
throttle: 10s # min interval between computes
timeout: 30s # max compute duration
gracePeriod: 5s # wait before first compute after startup
grace_period: 5s # wait before first compute after startup
reflexes:
- kind: sense
@@ -86,10 +91,6 @@ reflexes:
interval: 30s # periodic trigger
on: [disk-pressure] # also trigger on signals from other senses
- kind: workflow
workflow: cleanup
on: [disk-pressure] # start a workflow when signal fires
workflows:
cleanup:
concurrency: 1
@@ -97,43 +98,66 @@ workflows:
code-review:
concurrency: 3
overflow: queue
maxQueue: 20
max_queue: 20
```
YAML must **not** include `workflow:` under `reflexes` — the parser rejects it. Declare workflows under `workflows:` and start them from Sense `compute()` or `nerve workflow trigger`.
**Example — Sense starts a workflow** (`senses/disk-pressure/compute.ts`):
```typescript
export async function compute() {
const full = await diskNearlyFull();
if (!full) return null;
return {
path: "/data",
workflow: "cleanup|10|Disk partition nearly full", // name|maxRounds|prompt
};
}
```
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Kernel │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ │ Worker │ │ Worker │ │ Worker (1 per
│ (group A)│ │ (group B)│ │ (group C)│ group)
│ sense-1 │ │ sense-3 │ │ sense-5 │ │
│ │ sense-2 │ │ sense-4 │ │ │ │
│ └─────────┘ └────┬─────┘ └────┬─────┘
└──────────────┼──────────────┘
┌──────────────┐
│ Signal Bus │
│ └─────────────┘
──────────────────
│ Reflex Scheduler │
└────────┬─────────
┌───────────────────
Workflow Manager │──→ Log Store (SQLite)
└───────────────────
└─────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────
│ Kernel
│ ┌──────────────┐ watches nerve.yaml / senses / workflows
│ │ File Watcher ├──────────────────────────────────────────┐
└──────────────┘ │
┌──────────────┐ CLI ↔ newline JSON (trigger-workflow, │ │
│ │ Daemon IPC │ trigger-sense, list-senses) │ │
│ └──────┬───────┘
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ │ Worker │ │ Worker │ │ Worker │ (1 per
│ (group A)│ │ (group B)│ │ (group C)│ group)
│ sense-1 │ sense-3 │ sense-5 │
│ │ sense-2 │ │ sense-4 │ │
└────┬─────┘ └─────────└────┬─────┘
│ │
└────────────────────────────┘
│ ┌──────────────┐
│ Signal Bus │
│ └─────────────
│ │
│ ┌──────────────────
│ │ │ Reflex Scheduler│ │
│ │ └────────┬─────────┘ │
│ │ ▼ │
│ │ ┌───────────────────┐ │
│ └───────────────────►│ Workflow Manager │──→ @uncaged/nerve-store │
│ └───────────────────┘ (logs.db, …) │
└────────────────────────────────────────────────────────────────────────┘
```
- **Worker processes** — one per sense group, forked by the kernel. Isolated compute execution.
- **Worker pool** — one child process per sense group; isolation between groups.
- **Signal Bus** — in-memory pub/sub for signal distribution.
- **Reflex Scheduler** — interval timers + signal subscriptions, with throttle/coalesce.
- **Workflow Manager** — concurrency control (drop/queue), thread lifecycle tracking.
- **Log Store** — WAL-mode SQLite via `node:sqlite`, with archival and retention policies.
- **Workflow Manager** — concurrency (drop/queue), per-workflow workers, crash recovery.
- **File watcher** — hot reload for config, sense modules, and workflow modules.
- **Daemon IPC** — Unix domain socket; used by the CLI when the daemon is running.
- **Log / blob storage** — implemented in `@uncaged/nerve-store` (WAL SQLite, JSONL archive, CAS blobs).
## Tech Stack
+32 -14
View File
@@ -21,41 +21,59 @@ nerve init # Initialize a nerve workspace (installs deps, scaff
nerve validate # Validate nerve.yaml configuration
```
### Daemon Management
### Daemon management
```bash
nerve daemon start # Start the daemon (background)
nerve daemon stop # Stop the daemon
nerve daemon status # Check daemon health
nerve daemon restart # Restart the daemon
nerve daemon logs # Tail daemon logs
nerve daemon status # Show pid, uptime, sense names from nerve.yaml (process must exist)
nerve daemon restart # Stop then start
nerve daemon logs # Tail daemon process logs (file under workspace logs/)
```
### Development
```bash
nerve dev # Run in foreground mode (no daemon, Ctrl+C to stop)
nerve dev # Foreground kernel with file watcher + IPC (Ctrl+C stops)
```
### Querying
### Querying & status
```bash
nerve logs # View structured logs
nerve sense query <name> # Query a sense's SQLite database
nerve sense schema <name> # Show a sense's database schema
nerve status # Daemon health summary
nerve logs # Tail or page the daemon text log file (path in footer; default ~/.uncaged-nerve/logs/nerve.log)
nerve status # Short daemon health summary (aliases daemon status)
```
Structured rows in `data/logs.db` are surfaced via **`nerve workflow inspect`** / **`nerve workflow list`** (and `LogStore` in code), not via `nerve logs`.
### Sense
```bash
nerve sense list # List senses (live fields from daemon IPC when running)
nerve sense trigger <name> # IPC trigger-sense — queue a compute for that sense
nerve sense query <name> # Read-only SQL on data/senses/<name>.db (optional SQL args)
nerve sense schema <name> # Print CREATE TABLE statements for that sense DB
```
### Store maintenance
```bash
nerve store archive # Move old log rows to JSONL under data/archive/logs/… (optional --vacuum)
```
### Workflows
```bash
nerve workflow list # List workflow runs
nerve workflow show <runId> # Show workflow run details
nerve workflow list # Queued/started runs (add --all for terminal states; --workflow, --limit, --offset)
nerve workflow inspect <runId> # Run metadata + paginated workflow log lines
nerve workflow thread <runId> # Role rounds from persisted messages (--before, --budget)
nerve workflow trigger <name> # IPC trigger-workflow (daemon must be running)
# Optional JSON: --payload '{"prompt":"…","maxRounds":50}'
```
### Top-level Aliases
`nerve workflow trigger` sends a `trigger-workflow` line on the daemon Unix socket (same protocol as `@uncaged/nerve-core` / `parseDaemonIpcRequest`). It does not read `nerve.yaml` workflow definitions beyond what the running daemon already loaded.
For convenience, these aliases are available:
### Top-level aliases
```bash
nerve start → nerve daemon start
+29 -3
View File
@@ -4,9 +4,12 @@ Shared types and configuration parser for the [nerve](../../README.md) observati
## What's Inside
- **Type definitions** — `Signal`, `SenseConfig`, `ReflexConfig`, `WorkflowConfig`, `NerveConfig`, and all related types
- **Config parser** — `parseNerveConfig(yaml)` validates and parses `nerve.yaml` into a typed `NerveConfig`
- **Result type** — `Result<T>` with `ok()` / `err()` helpers for explicit error handling (no thrown exceptions)
- **Type definitions** — `Signal`, `SenseConfig`, `SenseInfo`, `SenseReflexConfig`, `ReflexConfig` (sense-only), `WorkflowConfig`, `NerveConfig`, and related types
- **Config parser** — `parseNerveConfig(yaml)` validates and parses `nerve.yaml` into `NerveConfig` (rejects reflex entries that declare a `workflow` key; reflexes only schedule senses)
- **Sense → workflow routing** — `parseSenseWorkflowDirective`, `routeSenseComputeOutput`, and types `ParsedSenseWorkflowDirective`, `SenseComputeRoute`
- **Daemon IPC protocol** — request/response types (`DaemonIpcRequest`, `DaemonIpcResponse`, …) and `parseDaemonIpcRequest` for newline-delimited JSON on the CLI ↔ daemon socket
- **Workflow automaton types** — `START` / `END` sentinel constants, `WorkflowMessage`, `StartSignal`, `RoleSignal`, `Moderator`, `WorkflowDefinition`, `Role`, `SenseResult`, plus `DEFAULT_ENGINE_MAX_ROUNDS`
- **Result type** — `Result<T>` with `ok()` / `err()` helpers for explicit error handling (no thrown exceptions for parse paths)
## Usage
@@ -20,6 +23,29 @@ if (result.ok) {
}
```
### Sense return → signal vs workflow
```typescript
import { parseSenseWorkflowDirective, routeSenseComputeOutput } from "@uncaged/nerve-core";
const directive = parseSenseWorkflowDirective("my-workflow|8|Hello from sense");
if (directive.ok) {
console.log(directive.value.workflowName, directive.value.maxRounds, directive.value.prompt);
}
const route = routeSenseComputeOutput({
metric: 42,
workflow: "my-workflow|8|Run now",
});
if (route.kind === "launch") {
// engine starts workflow; no Signal to the bus for this return
console.log(route.launch);
} else {
// normal signal with payload
console.log(route.payload);
}
```
## Duration Format
Config fields like `throttle`, `timeout`, and `interval` accept human-readable durations:
+53 -18
View File
@@ -4,18 +4,33 @@ The observation engine runtime for [nerve](../../README.md) — runs senses, rou
## Architecture
| Module | Responsibility |
|--------|---------------|
| **Kernel** | Top-level orchestrator — spawns workers, wires up signal bus, scheduler, and workflow manager. Supports hot reload and graceful shutdown. |
| **Sense Runtime** | Per-sense SQLite database (via `node:sqlite` + Drizzle ORM), migration runner, peer DB read access. |
| **Sense Worker** | Forked child process — one per sense group. Runs compute functions in isolation. |
| **Signal Bus** | In-memory pub/sub. Sense computes emit signals; reflexes and workflows subscribe. |
| **Reflex Scheduler** | Drives compute triggers — interval timers, signal-based events, throttle/coalesce logic. |
| **Workflow Manager** | Concurrency control (drop/queue), thread lifecycle, worker process management (RFC-002). |
| **Log Store** | Structured log storage in WAL-mode SQLite. Supports retention policies, archival to JSONL, and workflow run tracking. |
| **Blob Store** | Binary artifact storage for workflow outputs. |
| **File Watcher** | Watches `nerve.yaml` and sense files for hot reload. |
| **Daemon IPC** | Unix socket server for CLI ↔ daemon communication. |
| Module | Source (indicative) | Responsibility |
|--------|---------------------|----------------|
| **Kernel** | `kernel.ts` | Orchestrator — worker pool, signal bus, reflex scheduler, workflow manager, optional file watcher and daemon IPC, config reload hooks |
| **Worker pool** | `worker-pool.ts` | Fork and supervise one child process per sense group; restart/shutdown; crash cleanup hooks for scheduler state |
| **Kernel sense groups** | `kernel-sense-groups.ts` | Derive sense groups from config; list senses per group for scheduling |
| **Sense runtime** | sense worker + Drizzle | Per-sense SQLite (`node:sqlite`), migrations, peer DB reads |
| **Sense worker** | `sense-worker.ts` (fork target) | Child process entry — runs `compute()` per sense in a group |
| **Signal bus** | `signal-bus.ts` | In-memory pub/sub for sense signals |
| **Reflex scheduler** | `reflex-scheduler.ts` | Interval + `on` subscriptions, throttle/coalesce |
| **Workflow manager** | `workflow-manager.ts` | One worker per workflow name, concurrency (drop/queue), queue caps |
| **Workflow worker** | `workflow-worker.ts` | Child process — runs RFC-002 threads (`start-thread`, `resume-thread` IPC) |
| **IPC (parent ↔ workers)** | `ipc.ts` | Typed messages for sense and workflow workers (includes `resume-thread` for recovery) |
| **Log / workflow persistence** | via `@uncaged/nerve-store` | Structured logs, `workflow_runs`, thread messages (used for recovery) |
| **Blob store** | `@uncaged/nerve-store` | CAS under `data/blobs/` — sense workers construct `createBlobStore(join(nerveRoot, "data", "blobs"))` for artifact writes |
| **File watcher** | `file-watcher.ts` | Watches workspace paths for config / sense / workflow file changes |
| **Kernel file watch** | `kernel-file-watch.ts` | Maps watcher events to `reloadConfig`, sense group restart, workflow `drainAndRespawn` |
| **Daemon IPC** | `daemon-ipc.ts` | Unix socket server — parses `@uncaged/nerve-core` `DaemonIpcRequest`, dispatches trigger-workflow / trigger-sense / list-senses |
## Crash recovery (workflow workers)
If a workflow worker exits unexpectedly while threads are active:
- In-flight runs are marked **`crashed`** in the log store; the manager respawns a fresh worker.
- Runs still in **`started`** state can be **`resume-thread`**’d: the manager rebuilds the message chain from persisted workflow log rows and sends `resume-thread` to the new worker.
- **Crash-loop backoff:** repeated crashes for the same workflow name are counted in a sliding window (`60s`); after **`5`** crashes in that window, the manager **stops respawning** that worker and logs the condition (avoids tight crash loops).
Hot reload (`drainAndRespawn`) uses a controlled drain: in-flight runs may be marked **`interrupted`** when the old worker is torn down after a timeout — that path is distinct from unexpected crash recovery.
## Key Design Decisions
@@ -26,24 +41,44 @@ The observation engine runtime for [nerve](../../README.md) — runs senses, rou
## Usage
The daemon is typically started via the CLI (`nerve daemon start`), but can be used programmatically:
The daemon is typically started via the CLI (`nerve daemon start` / `nerve dev`), but you can embed the kernel:
```typescript
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { parseNerveConfig } from "@uncaged/nerve-core";
import { createKernel } from "@uncaged/nerve-daemon";
const kernel = await createKernel(nerveRoot);
const nerveRoot = "/path/to/workspace";
const yamlPath = join(nerveRoot, "nerve.yaml");
const parsed = parseNerveConfig(readFileSync(yamlPath, "utf8"));
if (!parsed.ok) {
throw parsed.error;
}
const kernel = createKernel(parsed.value, nerveRoot, {
enableFileWatcher: true,
ipcSocketPath: join(nerveRoot, "nerve.sock"),
});
await kernel.ready;
// Trigger a sense manually
kernel.triggerSense("cpu-usage");
// Check health
const health = kernel.getHealth();
// Graceful shutdown
await kernel.stop();
```
`createKernel(config, nerveRoot, options?)``config` is a parsed `NerveConfig`; `nerveRoot` is the workspace root (contains `nerve.yaml`, `data/`, etc.). Optional `KernelOptions`:
| Field | Meaning |
|-------|---------|
| `workerScript` | Override path to the sense worker entry script (defaults to the package’s resolved worker) |
| `enableFileWatcher` | Watch config / senses / workflows for hot reload |
| `logStore` | Inject a `LogStore` instance (defaults to `createLogStore(join(nerveRoot, "data", "logs.db"))`) |
| `ipcSocketPath` | When non-null, listen for daemon IPC on this Unix socket path |
## Install
```bash
+48
View File
@@ -0,0 +1,48 @@
# @uncaged/nerve-store
Persistent storage for the [nerve](../../README.md) daemon — append-only structured logs, optional JSONL cold archive, and content-addressable blobs.
## LogStore (`createLogStore`, `log-store.ts`)
- **Append-only log table** — rows with `source`, `type`, `refId`, `payload`, `ts` (string payloads for ad hoc fields)
- **SQLite WAL** — `DatabaseSync` from `node:sqlite`
- **Workflow run tracking** — materialized `workflow_runs` table plus helpers to list active runs, upsert status transitions, and read **thread messages** / **role rounds** for CLI and crash recovery
- **Meta key-value** — small `meta` table (e.g. archive watermarks)
Public exports include `LogStore`, `LogEntry`, `LogQuery`, `WorkflowRun`, `WorkflowRunStatus`, `ThreadRoundRow`, `GetThreadRoundsParams`, and archive-related types re-exported from `log-archive`.
## WorkflowRunStatus
Runs progress through a small state machine. Typical paths:
1. **`queued`** → **`started`** when a worker picks up the thread
2. **`started`** → **`completed`** | **`failed`** | **`crashed`** | **`interrupted`** | **`dropped`**
Semantics in the daemon/store layer:
- **`completed` / `failed`** — normal terminal outcomes from the workflow worker
- **`crashed`** — worker exited unexpectedly; manager may respawn and **`resume-thread`** eligible **`started`** runs
- **`interrupted`** — e.g. hot-reload drain killed an in-flight thread after timeout
- **`dropped`** — concurrency **`overflow: drop`** rejected a new run, or **`overflow: queue`** evicted an queued item when the queue was full
## LogArchive (`log-archive.ts`)
- **`archiveLogs`** / helpers — export eligible UTC days of old rows to **`data/archive/logs/YYYY-MM-DD.jsonl`**, delete archived rows from SQLite, optional **`VACUUM`**
- Used by **`nerve store archive`** in `@uncaged/nerve-cli`
## BlobStore (`createBlobStore`, `blob-store.ts`)
- **Content-addressable storage** — `write` returns lowercase **sha256** hex; files live under **`data/blobs/<2-hex>/<62-hex>`**
- **`read` / `exists`** — path must match digest on disk (tamper detection)
## Install
```bash
pnpm add @uncaged/nerve-store
```
Requires Node.js ≥ 22.5 (same as the rest of the stack).
## License
MIT