Compare commits

...

12 Commits

Author SHA1 Message Date
xiaomo 5be14d0d8b docs: add comprehensive README for root and all packages 2026-04-23 10:53:45 +00:00
xiaomo cf2b0ac223 Merge pull request 'build: migrate from tsup to rslib' (#71) from build/tsup-to-rslib into main 2026-04-23 09:50:55 +00:00
xiaoju 1b5a52ea4d build: migrate from tsup to rslib
Replace tsup (esbuild-based) with rslib (rspack-based) across all packages.

tsup's built-in nodeProtocolPlugin strips the 'node:' prefix from all
Node.js builtins. Unlike node:fs etc., node:sqlite has no unprefixed
form, causing ERR_MODULE_NOT_FOUND at runtime. rslib handles node:
imports correctly without any workarounds.

Changes:
- Replace tsup.config.ts with rslib.config.ts in core, daemon, cli
- Swap tsup → @rslib/core in devDependencies
- Fix log-store.ts params type (Record<string, unknown> → Record<string, string | number>)
- Fix logStream.fd type cast in start.ts
- Exclude __tests__ from CLI tsconfig to avoid DTS errors
- All 356 tests pass, nerve init works correctly

Closes #70

小橘 🍊(NEKO Team)
2026-04-23 09:48:45 +00:00
xiaoju a084205b47 Revert "fix: restore node:sqlite prefix stripped by tsup bundler"
This reverts commit 57550ccfdb.
2026-04-23 09:41:28 +00:00
xiaoju 57550ccfdb fix: restore node:sqlite prefix stripped by tsup bundler
tsup's built-in node-protocol-plugin strips the 'node:' prefix from
all builtins. Unlike node:fs etc., node:sqlite has no unprefixed form,
causing ERR_MODULE_NOT_FOUND at runtime.

- Add onSuccess hook to both cli and daemon tsup configs to restore
  'node:sqlite' imports in bundled output
- Fix log-store params type to Record<string, string | number>

小橘 🍊(NEKO Team)
2026-04-23 09:32:20 +00:00
xiaomo 37588df402 Merge pull request 'refactor(daemon): upgrade Drizzle v1.0-beta + migrate better-sqlite3 → node:sqlite' (#69) from refactor/drizzle-v1-node-sqlite into main 2026-04-23 09:20:15 +00:00
xiaoju 85dd11c84d refactor(daemon): upgrade Drizzle v1.0-beta + migrate better-sqlite3 → node:sqlite
- Upgrade drizzle-orm from 0.43.1 to 1.0.0-beta.23
- Replace better-sqlite3 with node:sqlite (DatabaseSync) in:
  - sense-runtime.ts (Drizzle driver)
  - log-store.ts (raw SQL)
  - all test files
- Replace sqlite.pragma() with sqlite.exec('PRAGMA ...')
- Replace sqlite.transaction() with manual BEGIN/COMMIT/ROLLBACK
- Update CLI init command to verify node:sqlite instead of better-sqlite3
- Remove better-sqlite3 and @types/better-sqlite3 from dependencies
- Zero native addons remaining in the monorepo 🎉

Closes #67

小橘 <xiaoju@shazhou.work>
2026-04-23 09:18:44 +00:00
xiaomo d80a414530 Merge pull request 'chore: walkthrough cleanup — engines, types, mock fixes' (#68) from fix/walkthrough-cleanup into main 2026-04-23 09:10:09 +00:00
xiaoju 7f780f0642 chore: walkthrough cleanup — engines, types, mock fixes
- Add engines >= 22.5.0 to root and cli package.json (node:sqlite requirement)
- Remove unused @types/better-sqlite3 from cli devDeps (leftover from sql.js migration)
- Add files/publishConfig to core package.json (parity with other packages)
- Fix daemon test type errors: add getAllWorkflowRuns to mock LogStore,
  fix array destructuring on mock.calls, fix sense-runtime callback signatures

All 356 tests pass across all packages.

小橘 🍊(NEKO Team)
2026-04-23 09:08:24 +00:00
xiaomo 33e0d9a705 Merge pull request 'refactor(cli): replace sql.js with node:sqlite' (#66) from refactor/node-sqlite into main 2026-04-23 08:51:01 +00:00
xiaoju 418d8ee0c8 refactor(cli): replace sql.js with node:sqlite
Drop the sql.js WASM dependency in favour of Node 22's built-in
node:sqlite (DatabaseSync). This eliminates the ~2 MB WASM binary,
removes the async init ceremony, and lets us open databases in
readonly mode directly on disk instead of loading them into memory.

Breaking: requires Node >= 22.5.0 (sqlite support).

- Remove sql.js from cli dependencies
- Rewrite sense-sqlite.ts to use DatabaseSync
- Update sense command (schema/query) — sync API, no more queryAsObjects
- Update tests to use node:sqlite directly
- Remove sql.js from tsup externals

小橘 🍊(NEKO Team)
2026-04-23 08:43:39 +00:00
xiaomo 719c4c1449 Merge pull request 'refactor(cli): replace better-sqlite3 with sql.js (pure WASM) — implements RFC #63' (#64) from refactor/sql-js-migration into main 2026-04-23 07:32:38 +00:00
30 changed files with 1641 additions and 770 deletions
+163 -1
View File
@@ -1,3 +1,165 @@
# nerve
Observation engine — Sense, Reflex, Workflow
**Observation engine for autonomous agents**sense the world, react to changes, run workflows.
Nerve is a lightweight daemon that continuously observes external state through **Senses**, reacts via declarative **Reflexes**, and orchestrates multi-step **Workflows**. Built for the [Uncaged](https://github.com/uncaged) agent framework.
## Core Concepts
```
External World → Sense → Signal → Reflex → Workflow → Log
↑ ↑
"what to observe" "what to do"
```
| Concept | Metaphor | Role |
|---------|----------|------|
| **Sense** | 👁️ Perception | A `compute()` function that samples or derives data. Each sense has its own SQLite database. |
| **Reflex** | ⚡ Reaction | Declarative trigger — interval-based, event-driven, or both. Connects senses to actions. |
| **Signal** | 📡 Notification | Emitted when a sense returns non-null. Other reflexes can listen for signals. |
| **Workflow** | 🔧 Action | Stateful multi-step execution with Roles (actors) and a Moderator (coordinator). |
| **Log** | 📝 Record | Immutable audit trail. Queryable by senses, but **cannot** trigger reflexes (prevents feedback loops). |
Three extension points, fully orthogonal — a Sense doesn't know when it runs, a Reflex doesn't know what it computes, a Workflow doesn't know why it was triggered.
## Packages
| Package | Description |
|---------|-------------|
| [`@uncaged/nerve-core`](./packages/core) | Shared types and config parser |
| [`@uncaged/nerve-daemon`](./packages/daemon) | The observation engine — kernel, sense runtime, reflex scheduler, workflow manager |
| [`@uncaged/nerve-cli`](./packages/cli) | CLI tool (`nerve`) — init, start, stop, logs, query |
## Quick Start
```bash
# Requirements: Node.js ≥ 22.5, pnpm
pnpm add -g @uncaged/nerve-cli
# Initialize a workspace
mkdir my-agent && cd my-agent
nerve init
# Write a sense
cat > senses/cpu-usage/compute.ts << 'EOF'
export async function compute() {
const [load] = (await import("node:os")).loadavg();
return load > 2.0 ? { load } : null; // signal only when load is high
}
EOF
# Configure reflexes in nerve.yaml
cat > nerve.yaml << 'EOF'
senses:
cpu-usage:
group: system
throttle: 10s
reflexes:
- kind: sense
sense: cpu-usage
interval: 30s
EOF
# Run
nerve dev # foreground (development)
nerve daemon start # background (production)
nerve status # check health
nerve logs # view logs
```
## Configuration
`nerve.yaml` declares senses, reflexes, and workflows:
```yaml
senses:
cpu-usage:
group: system # senses in the same group share a worker process
throttle: 10s # min interval between computes
timeout: 30s # max compute duration
gracePeriod: 5s # wait before first compute after startup
reflexes:
- kind: sense
sense: cpu-usage
interval: 30s # periodic trigger
on: [disk-pressure] # also trigger on signals from other senses
- kind: workflow
workflow: cleanup
on: [disk-pressure] # start a workflow when signal fires
workflows:
cleanup:
concurrency: 1
overflow: drop # discard if already running
code-review:
concurrency: 3
overflow: queue
maxQueue: 20
```
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Kernel │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Worker │ │ Worker │ │ Worker │ (1 per │
│ │ (group A)│ │ (group B)│ │ (group C)│ group) │
│ │ sense-1 │ │ sense-3 │ │ sense-5 │ │
│ │ sense-2 │ │ sense-4 │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ └──────────────┼──────────────┘ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Signal Bus │ │
│ └──────┬───────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Reflex Scheduler │ │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ Workflow Manager │──→ Log Store (SQLite) │
│ └───────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
- **Worker processes** — one per sense group, forked by the kernel. Isolated compute execution.
- **Signal Bus** — in-memory pub/sub for signal distribution.
- **Reflex Scheduler** — interval timers + signal subscriptions, with throttle/coalesce.
- **Workflow Manager** — concurrency control (drop/queue), thread lifecycle tracking.
- **Log Store** — WAL-mode SQLite via `node:sqlite`, with archival and retention policies.
## Tech Stack
- **Zero native addons** — uses Node.js built-in `node:sqlite` (DatabaseSync)
- **Drizzle ORM** v1.0 for sense databases
- **rslib** (rspack) for building
- **Biome** for formatting/linting
- **Vitest** for testing
- **pnpm** workspaces for monorepo management
## Development
```bash
git clone https://git.shazhou.work/uncaged/nerve.git
cd nerve
pnpm install
pnpm build
pnpm -r test # run all tests
```
## Design Documents
- [RFC-001: Observation Engine](./docs/rfc-001-observation-engine.md) — Sense, Signal, Reflex model
- [RFC-002: Workflow Engine](./docs/rfc-002-workflow-engine.md) — Stateful workflow execution
- [Coding Conventions](./docs/coding-conventions.md)
## License
MIT
+4 -1
View File
@@ -1,6 +1,9 @@
{
"name": "nerve",
"private": true,
"engines": {
"node": ">=22.5.0"
},
"scripts": {
"build": "pnpm -r run build",
"check": "biome check .",
@@ -8,7 +11,7 @@
},
"devDependencies": {
"@biomejs/biome": "^1.9.0",
"tsup": "^8.0.0",
"@rslib/core": "^0.21.3",
"typescript": "^5.5.0"
}
}
+69
View File
@@ -0,0 +1,69 @@
# @uncaged/nerve-cli
Command-line interface for the [nerve](../../README.md) observation engine.
## Install
```bash
pnpm add -g @uncaged/nerve-cli
# or
npx @uncaged/nerve-cli
```
Requires Node.js ≥ 22.5.
## Commands
### Workspace
```bash
nerve init # Initialize a nerve workspace (installs deps, scaffolds config)
nerve validate # Validate nerve.yaml configuration
```
### Daemon Management
```bash
nerve daemon start # Start the daemon (background)
nerve daemon stop # Stop the daemon
nerve daemon status # Check daemon health
nerve daemon restart # Restart the daemon
nerve daemon logs # Tail daemon logs
```
### Development
```bash
nerve dev # Run in foreground mode (no daemon, Ctrl+C to stop)
```
### Querying
```bash
nerve logs # View structured logs
nerve sense query <name> # Query a sense's SQLite database
nerve sense schema <name> # Show a sense's database schema
nerve status # Daemon health summary
```
### Workflows
```bash
nerve workflow list # List workflow runs
nerve workflow show <runId> # Show workflow run details
```
### Top-level Aliases
For convenience, these aliases are available:
```bash
nerve start → nerve daemon start
nerve stop → nerve daemon stop
nerve status → nerve daemon status
nerve logs → nerve daemon logs
```
## License
MIT
+6 -4
View File
@@ -1,5 +1,8 @@
{
"name": "@uncaged/nerve-cli",
"engines": {
"node": ">=22.5.0"
},
"version": "0.1.8",
"type": "module",
"bin": {
@@ -15,16 +18,15 @@
},
"scripts": {
"prepublishOnly": "bash ../../scripts/prepublish-check.sh",
"build": "tsup",
"build": "rslib build",
"test": "vitest run"
},
"dependencies": {
"@uncaged/nerve-core": "workspace:*",
"citty": "^0.1.6",
"sql.js": "^1.14.1"
"citty": "^0.1.6"
},
"devDependencies": {
"@types/better-sqlite3": "^7.6.13",
"@rslib/core": "^0.21.3",
"@types/node": "^22.0.0",
"@uncaged/nerve-daemon": "workspace:*",
"vitest": "^4.1.5"
+25
View File
@@ -0,0 +1,25 @@
import { defineConfig } from "@rslib/core";
export default defineConfig({
lib: [
{
format: "esm",
dts: true,
banner: {
js: "#!/usr/bin/env node",
},
},
],
source: {
entry: {
index: "src/index.ts",
cli: "src/cli.ts",
"daemon-bootstrap": "src/daemon-bootstrap.ts",
},
},
output: {
target: "node",
cleanDistPath: true,
externals: ["@uncaged/nerve-daemon"],
},
});
+28 -51
View File
@@ -2,12 +2,12 @@
* Tests for sense SQLite helpers used by `nerve sense schema` / `nerve sense query`.
*/
import { mkdirSync, rmSync, writeFileSync } from "node:fs";
import { mkdirSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { DatabaseSync } from "node:sqlite";
import initSqlJs, { type Database } from "sql.js";
import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
assertSenseDbExists,
@@ -17,17 +17,11 @@ import {
listTableSqlStatements,
parseSenseQueryArgs,
pickDefaultPreviewTable,
queryAsObjects,
senseDbPath,
} from "../sense-sqlite.js";
let SQL: Awaited<ReturnType<typeof initSqlJs>>;
let tmpDir: string;
beforeAll(async () => {
SQL = await initSqlJs();
});
beforeEach(() => {
tmpDir = join(
tmpdir(),
@@ -40,22 +34,6 @@ afterEach(() => {
rmSync(tmpDir, { recursive: true, force: true });
});
/** Helper: create a SQLite db file with the given setup SQL. */
function createDb(name: string, setupSql: string): void {
const db = new SQL.Database();
db.run(setupSql);
const data = db.export();
db.close();
writeFileSync(join(tmpDir, "data", "senses", `${name}.db`), Buffer.from(data));
}
/** Helper: open an in-memory db with setup SQL for unit tests. */
function memDb(setupSql?: string): Database {
const db = new SQL.Database();
if (setupSql) db.run(setupSql);
return db;
}
describe("senseDbPath", () => {
it("points at data/senses/<name>.db under the given root", () => {
expect(senseDbPath("/root", "cpu-usage")).toBe(join("/root", "data", "senses", "cpu-usage.db"));
@@ -68,14 +46,18 @@ describe("assertSenseDbExists", () => {
});
it("returns the path when the file exists", () => {
createDb("x", "SELECT 1");
expect(assertSenseDbExists(tmpDir, "x")).toBe(join(tmpDir, "data", "senses", "x.db"));
const p = join(tmpDir, "data", "senses", "x.db");
new DatabaseSync(p).close();
expect(assertSenseDbExists(tmpDir, "x")).toBe(p);
});
});
describe("listTableSqlStatements", () => {
it("returns CREATE statements ordered by tbl_name", () => {
const db = memDb("CREATE TABLE zebra (id INTEGER); CREATE TABLE alpha (id INTEGER);");
const p = join(tmpDir, "data", "senses", "t.db");
const db = new DatabaseSync(p);
db.exec("CREATE TABLE zebra (id INTEGER)");
db.exec("CREATE TABLE alpha (id INTEGER)");
const stmts = listTableSqlStatements(db);
db.close();
expect(stmts).toHaveLength(2);
@@ -86,16 +68,18 @@ describe("listTableSqlStatements", () => {
describe("pickDefaultPreviewTable", () => {
it("prefers non-_migrations tables when both exist", () => {
const db = memDb(
`CREATE TABLE _migrations (name TEXT PRIMARY KEY);
CREATE TABLE readings (id INTEGER);`,
);
const p = join(tmpDir, "data", "senses", "t.db");
const db = new DatabaseSync(p);
db.exec("CREATE TABLE _migrations (name TEXT PRIMARY KEY)");
db.exec("CREATE TABLE readings (id INTEGER)");
expect(pickDefaultPreviewTable(db)).toBe("readings");
db.close();
});
it("uses _migrations when it is the only table", () => {
const db = memDb("CREATE TABLE _migrations (name TEXT PRIMARY KEY);");
const p = join(tmpDir, "data", "senses", "t.db");
const db = new DatabaseSync(p);
db.exec("CREATE TABLE _migrations (name TEXT PRIMARY KEY)");
expect(pickDefaultPreviewTable(db)).toBe("_migrations");
db.close();
});
@@ -153,29 +137,22 @@ describe("collectColumnKeys", () => {
});
});
describe("queryAsObjects", () => {
it("converts columnar sql.js results to row objects", () => {
const db = memDb("CREATE TABLE t (x INTEGER, y TEXT); INSERT INTO t VALUES (1, 'a'), (2, 'b');");
const rows = queryAsObjects(db, "SELECT * FROM t ORDER BY x");
db.close();
expect(rows).toEqual([
{ x: 1, y: "a" },
{ x: 2, y: "b" },
]);
});
});
describe("readonly query integration", () => {
it("runs default preview SQL on a real db file", () => {
createDb("demo", "CREATE TABLE items (id INTEGER PRIMARY KEY, v TEXT); INSERT INTO items (v) VALUES ('a'), ('b');");
it("runs default preview SQL on a real db", () => {
const p = join(tmpDir, "data", "senses", "demo.db");
const rw = new DatabaseSync(p);
rw.exec("CREATE TABLE items (id INTEGER PRIMARY KEY, v TEXT)");
rw.exec("INSERT INTO items (v) VALUES ('a'), ('b')");
rw.close();
const buffer = require("node:fs").readFileSync(join(tmpDir, "data", "senses", "demo.db"));
const db = new SQL.Database(buffer);
const db = new DatabaseSync(p, { readOnly: true });
const table = pickDefaultPreviewTable(db);
expect(table).toBe("items");
if (table === null) throw new Error("expected items table");
if (table === null) {
throw new Error("expected items table");
}
const sql = defaultPreviewSql(table);
const rows = queryAsObjects(db, sql);
const rows = db.prepare(sql).all() as Record<string, unknown>[];
db.close();
expect(rows.length).toBeGreaterThanOrEqual(1);
});
+18 -30
View File
@@ -1,4 +1,4 @@
import { spawn, execFile } from "node:child_process";
import { execFile, spawn } from "node:child_process";
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { promisify } from "node:util";
@@ -35,7 +35,7 @@ const PACKAGE_JSON = `{
"drizzle-kit": "latest"
},
"pnpm": {
"onlyBuiltDependencies": ["better-sqlite3", "esbuild"]
"onlyBuiltDependencies": ["esbuild"]
}
}
`;
@@ -218,14 +218,18 @@ const initWorkspaceCommand = defineCommand({
},
});
async function tryRequireSqlite(nerveRoot: string): Promise<boolean> {
/** Verify built-in `node:sqlite` (Node.js ≥22.5) loads in a child process. */
async function verifyNodeSqlite(): Promise<boolean> {
try {
const modulePath = join(nerveRoot, "node_modules", "better-sqlite3");
// Use a child process to test if the native module loads
await execFileAsync("node", ["-e", `require(${JSON.stringify(modulePath)})`], {
cwd: nerveRoot,
timeout: 10_000,
});
await execFileAsync(
"node",
[
"--input-type=module",
"-e",
"import { DatabaseSync } from 'node:sqlite'; new DatabaseSync(':memory:').exec('SELECT 1');",
],
{ timeout: 10_000 },
);
return true;
} catch {
return false;
@@ -264,27 +268,11 @@ async function runInitWorkspace(force: boolean): Promise<void> {
);
}
// Verify better-sqlite3 native module — rebuild up to 2 times if broken
const sqlitePath = join(nerveRoot, "node_modules", "better-sqlite3");
if (existsSync(sqlitePath)) {
for (let attempt = 1; attempt <= 2; attempt++) {
if (await tryRequireSqlite(nerveRoot)) break;
process.stdout.write(
`${attempt === 1 ? "Building" : "Retrying build of"} native module better-sqlite3 (attempt ${attempt}/2)…\n`,
);
try {
await runCommand(cmd, ["rebuild", "better-sqlite3"], nerveRoot);
} catch {
// will be caught by the verify below
}
}
if (!(await tryRequireSqlite(nerveRoot))) {
process.stdout.write(
`⚠️ better-sqlite3 native module is not working. The daemon will fail to start.\n` +
` Fix: cd ${nerveRoot} && ${cmd} rebuild better-sqlite3\n` +
` Or: npm install --build-from-source better-sqlite3\n`,
);
}
if (!(await verifyNodeSqlite())) {
process.stdout.write(
"⚠️ Built-in SQLite (node:sqlite) is not available in this Node.js build. " +
"The daemon requires Node.js 22.5 or newer with SQLite enabled.\n",
);
}
if (!existsSync(join(nerveRoot, ".git"))) {
+6 -7
View File
@@ -1,5 +1,6 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { DatabaseSync } from "node:sqlite";
import { type SenseInfo, parseNerveConfig } from "@uncaged/nerve-core";
import { defineCommand } from "citty";
@@ -13,7 +14,6 @@ import {
openSenseDb,
parseSenseQueryArgs,
pickDefaultPreviewTable,
queryAsObjects,
} from "../sense-sqlite.js";
import { getNerveRoot, getSocketPath, isRunning } from "../workspace.js";
@@ -80,7 +80,6 @@ const senseListCommand = defineCommand({
},
async run() {
if (!isRunning()) {
// Daemon not running — show static info from nerve.yaml
process.stderr.write(
"⚠️ Daemon is not running — showing static config only (no last signal time).\n\n",
);
@@ -171,9 +170,9 @@ const senseSchemaCommand = defineCommand({
},
async run({ args }) {
const nerveRoot = getNerveRoot();
let db: ReturnType<Awaited<ReturnType<typeof import("sql.js")>>["Database"]> | undefined;
let db: DatabaseSync | undefined;
try {
db = await openSenseDb(nerveRoot, args.name);
db = openSenseDb(nerveRoot, args.name);
const statements = listTableSqlStatements(db);
if (args.json) {
process.stdout.write(`${JSON.stringify(statements, null, 2)}\n`);
@@ -217,7 +216,7 @@ const senseQueryCommand = defineCommand({
},
async run({ args, rawArgs }) {
const nerveRoot = getNerveRoot();
let db: ReturnType<Awaited<ReturnType<typeof import("sql.js")>>["Database"]> | undefined;
let db: DatabaseSync | undefined;
try {
let parsed: { name: string; sql: string | undefined };
try {
@@ -228,7 +227,7 @@ const senseQueryCommand = defineCommand({
process.exit(1);
}
db = await openSenseDb(nerveRoot, args.name);
db = openSenseDb(nerveRoot, args.name);
let sql = parsed.sql?.trim();
if (!sql) {
@@ -241,7 +240,7 @@ const senseQueryCommand = defineCommand({
}
}
const rows = queryAsObjects(db, sql);
const rows = db.prepare(sql).all() as Record<string, unknown>[];
if (args.json) {
process.stdout.write(`${JSON.stringify(rows, null, 2)}\n`);
+1 -1
View File
@@ -76,7 +76,7 @@ async function runDaemon(nerveRoot: string): Promise<void> {
const child = spawn(process.execPath, [bootstrapPath], {
detached: true,
stdio: ["ignore", logStream.fd, logStream.fd],
stdio: ["ignore", (logStream as any).fd, (logStream as any).fd],
env: { ...process.env, NERVE_ROOT: nerveRoot },
cwd: nerveRoot,
});
+29 -57
View File
@@ -1,25 +1,6 @@
import { existsSync, readFileSync } from "node:fs";
import { existsSync } from "node:fs";
import { join } from "node:path";
import initSqlJs, { type Database } from "sql.js";
// ── WASM singleton ──────────────────────────────────────────────────────────
let _SQL: Awaited<ReturnType<typeof initSqlJs>> | null = null;
async function getSQL() {
if (!_SQL) {
_SQL = await initSqlJs();
}
return _SQL;
}
/** Open a sense SQLite database (readonly, loaded into memory via sql.js). */
export async function openSenseDb(nerveRoot: string, senseName: string): Promise<Database> {
const path = assertSenseDbExists(nerveRoot, senseName);
const SQL = await getSQL();
const buffer = readFileSync(path);
return new SQL.Database(buffer);
}
import { DatabaseSync } from "node:sqlite";
/** SQLite path for a sense under the nerve workspace root. */
export function senseDbPath(nerveRoot: string, senseName: string): string {
@@ -34,31 +15,39 @@ export function assertSenseDbExists(nerveRoot: string, senseName: string): strin
return path;
}
/** Open a sense SQLite database in readonly mode using node:sqlite. */
export function openSenseDb(nerveRoot: string, senseName: string): DatabaseSync {
const path = assertSenseDbExists(nerveRoot, senseName);
return new DatabaseSync(path, { readOnly: true });
}
/** `SELECT sql FROM sqlite_master WHERE type='table'` (non-null sql only). */
export function listTableSqlStatements(db: Database): string[] {
const results = db.exec(
`SELECT sql FROM sqlite_master WHERE type = 'table' AND sql IS NOT NULL ORDER BY tbl_name`,
);
if (results.length === 0) return [];
return results[0].values.map((row) => row[0] as string);
export function listTableSqlStatements(db: DatabaseSync): string[] {
const rows = db
.prepare(
`SELECT sql FROM sqlite_master WHERE type = 'table' AND sql IS NOT NULL ORDER BY tbl_name`,
)
.all() as { sql: string }[];
return rows.map((r) => r.sql);
}
/**
* Table used for `nerve sense query <name>` with no SQL.
* Prefers real data tables over `_migrations`, then lexicographic by name.
*/
export function pickDefaultPreviewTable(db: Database): string | null {
const results = db.exec(
`SELECT name FROM sqlite_master
WHERE type = 'table' AND sql IS NOT NULL
AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
ORDER BY
CASE WHEN name = '_migrations' THEN 1 ELSE 0 END,
name
LIMIT 1`,
);
if (results.length === 0 || results[0].values.length === 0) return null;
return results[0].values[0][0] as string;
export function pickDefaultPreviewTable(db: DatabaseSync): string | null {
const row = db
.prepare(
`SELECT name FROM sqlite_master
WHERE type = 'table' AND sql IS NOT NULL
AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
ORDER BY
CASE WHEN name = '_migrations' THEN 1 ELSE 0 END,
name
LIMIT 1`,
)
.get() as { name: string } | undefined;
return row?.name ?? null;
}
export function defaultPreviewSql(table: string): string {
@@ -93,7 +82,7 @@ function stringifyCell(value: unknown): string {
if (typeof value === "bigint") return value.toString();
if (typeof value === "number" || typeof value === "boolean") return String(value);
if (typeof value === "string") return value;
if (value instanceof Uint8Array) return Buffer.from(value).toString("hex");
if (Buffer.isBuffer(value)) return value.toString("hex");
try {
return JSON.stringify(value);
} catch {
@@ -136,20 +125,3 @@ export function formatRowsAsAlignedTable(rows: Record<string, unknown>[]): strin
const body = cells.map((r) => r.map((cell, j) => cell.padEnd(widths[j])).join(" | ")).join("\n");
return `${header}\n${sep}\n${body}\n`;
}
/**
* Run a SQL query via sql.js and return rows as key-value objects.
* sql.js returns columnar data; this converts to the familiar row format.
*/
export function queryAsObjects(db: Database, sql: string): Record<string, unknown>[] {
const results = db.exec(sql);
if (results.length === 0) return [];
const { columns, values } = results[0];
return values.map((row) => {
const obj: Record<string, unknown> = {};
for (let i = 0; i < columns.length; i++) {
obj[columns[i]] = row[i];
}
return obj;
});
}
+2 -1
View File
@@ -6,5 +6,6 @@
"composite": false,
"types": ["node"]
},
"include": ["src"]
"include": ["src"],
"exclude": ["src/__tests__"]
}
-13
View File
@@ -1,13 +0,0 @@
import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts", "src/cli.ts", "src/daemon-bootstrap.ts"],
format: ["esm"],
dts: true,
clean: true,
banner: {
js: "#!/usr/bin/env node",
},
/** Daemon is loaded from workspace node_modules at runtime — never bundle it. */
external: ["@uncaged/nerve-daemon", "sql.js"],
});
+39
View File
@@ -0,0 +1,39 @@
# @uncaged/nerve-core
Shared types and configuration parser for the [nerve](../../README.md) observation engine.
## What's Inside
- **Type definitions** — `Signal`, `SenseConfig`, `ReflexConfig`, `WorkflowConfig`, `NerveConfig`, and all related types
- **Config parser** — `parseNerveConfig(yaml)` validates and parses `nerve.yaml` into a typed `NerveConfig`
- **Result type** — `Result<T>` with `ok()` / `err()` helpers for explicit error handling (no thrown exceptions)
## Usage
```typescript
import { parseNerveConfig, ok, err } from "@uncaged/nerve-core";
import type { NerveConfig, Signal, Result } from "@uncaged/nerve-core";
const result: Result<NerveConfig> = parseNerveConfig(yamlString);
if (result.ok) {
console.log(result.value.senses);
}
```
## Duration Format
Config fields like `throttle`, `timeout`, and `interval` accept human-readable durations:
- `5s` — 5 seconds
- `10m` — 10 minutes
- `1h` — 1 hour
## Install
```bash
pnpm add @uncaged/nerve-core
```
## License
MIT
+8 -1
View File
@@ -3,16 +3,23 @@
"version": "0.1.4",
"type": "module",
"main": "dist/index.js",
"files": [
"dist"
],
"publishConfig": {
"access": "public"
},
"types": "dist/index.d.ts",
"scripts": {
"prepublishOnly": "bash ../../scripts/prepublish-check.sh",
"build": "tsup",
"build": "rslib build",
"test": "vitest run"
},
"dependencies": {
"yaml": "^2.8.3"
},
"devDependencies": {
"@rslib/core": "^0.21.3",
"vitest": "^4.1.5"
}
}
+19
View File
@@ -0,0 +1,19 @@
import { defineConfig } from "@rslib/core";
export default defineConfig({
lib: [
{
format: "esm",
dts: true,
},
],
source: {
entry: {
index: "src/index.ts",
},
},
output: {
target: "node",
cleanDistPath: true,
},
});
-8
View File
@@ -1,8 +0,0 @@
import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts"],
format: ["esm"],
dts: true,
clean: true,
});
+57
View File
@@ -0,0 +1,57 @@
# @uncaged/nerve-daemon
The observation engine runtime for [nerve](../../README.md) — runs senses, routes signals, schedules reflexes, and manages workflows.
## Architecture
| Module | Responsibility |
|--------|---------------|
| **Kernel** | Top-level orchestrator — spawns workers, wires up signal bus, scheduler, and workflow manager. Supports hot reload and graceful shutdown. |
| **Sense Runtime** | Per-sense SQLite database (via `node:sqlite` + Drizzle ORM), migration runner, peer DB read access. |
| **Sense Worker** | Forked child process — one per sense group. Runs compute functions in isolation. |
| **Signal Bus** | In-memory pub/sub. Sense computes emit signals; reflexes and workflows subscribe. |
| **Reflex Scheduler** | Drives compute triggers — interval timers, signal-based events, throttle/coalesce logic. |
| **Workflow Manager** | Concurrency control (drop/queue), thread lifecycle, worker process management (RFC-002). |
| **Log Store** | Structured log storage in WAL-mode SQLite. Supports retention policies, archival to JSONL, and workflow run tracking. |
| **Blob Store** | Binary artifact storage for workflow outputs. |
| **File Watcher** | Watches `nerve.yaml` and sense files for hot reload. |
| **Daemon IPC** | Unix socket server for CLI ↔ daemon communication. |
## Key Design Decisions
- **One worker process per sense group** — isolation between groups, shared compute within a group
- **`node:sqlite` (DatabaseSync)** — zero native addons, WAL mode, built into Node.js ≥ 22.5
- **Throttle + coalesce** — if compute is in-flight, at most one pending trigger is queued (no unbounded accumulation)
- **Log ≠ Signal** — logs are queryable data assets but cannot trigger reflexes (prevents feedback loops)
## Usage
The daemon is typically started via the CLI (`nerve daemon start`), but can be used programmatically:
```typescript
import { createKernel } from "@uncaged/nerve-daemon";
const kernel = await createKernel(nerveRoot);
await kernel.ready;
// Trigger a sense manually
kernel.triggerSense("cpu-usage");
// Check health
const health = kernel.getHealth();
// Graceful shutdown
await kernel.stop();
```
## Install
```bash
pnpm add @uncaged/nerve-daemon
```
Requires Node.js ≥ 22.5 (for `node:sqlite`).
## License
MIT
+4 -4
View File
@@ -12,17 +12,17 @@
},
"scripts": {
"prepublishOnly": "bash ../../scripts/prepublish-check.sh",
"build": "tsup",
"build": "rslib build",
"test": "vitest run"
},
"dependencies": {
"@uncaged/nerve-core": "workspace:*",
"better-sqlite3": "^11.10.0",
"drizzle-orm": "^0.43.1",
"drizzle-orm": "1.0.0-beta.23-c10d10c",
"yaml": "^2.8.3"
},
"devDependencies": {
"@types/better-sqlite3": "^7.6.13",
"@rslib/core": "^0.21.3",
"@types/node": "^22.0.0",
"vitest": "^4.1.5"
}
}
+20
View File
@@ -0,0 +1,20 @@
import { defineConfig } from "@rslib/core";
export default defineConfig({
lib: [
{
format: "esm",
dts: true,
},
],
source: {
entry: {
index: "src/index.ts",
"sense-worker": "src/sense-worker.ts",
},
},
output: {
target: "node",
cleanDistPath: true,
},
});
@@ -89,10 +89,11 @@ function makeLogStore(
}
return activeRuns;
}),
getTriggerPayload: vi.fn(() => ({ value: 42 })),
getThreadEvents: vi.fn(() => [{ type: "thread_start", triggerPayload: {} }]),
getTriggerPayload: vi.fn((): unknown => ({ value: 42 })),
getThreadEvents: vi.fn((): Array<{ type: string; [key: string]: unknown }> => [{ type: "thread_start", triggerPayload: {} }]),
archiveLogs: vi.fn(() => ({ days: [], vacuumed: false })),
close: vi.fn(),
getAllWorkflowRuns: vi.fn(() => []),
};
return store;
}
@@ -127,7 +128,7 @@ describe("WorkflowManager — crash recovery (Phase 3)", () => {
child.emit("exit", 1, null);
const crashedCalls = logStore.upsertWorkflowRun.mock.calls.filter(
([entry]: [{ type: string }]) => entry.type === "crashed",
(args: any[]) => (args[0] as { type: string }).type === "crashed",
);
expect(crashedCalls).toHaveLength(2);
@@ -216,10 +217,10 @@ describe("WorkflowManager — crash recovery (Phase 3)", () => {
// resume-thread should have been sent
const resumeCalls = (secondChild.send as ReturnType<typeof vi.fn>).mock.calls.filter(
([msg]: [unknown]) =>
msg !== null &&
typeof msg === "object" &&
(msg as Record<string, unknown>).type === "resume-thread",
(args: any[]) =>
args[0] !== null &&
typeof args[0] === "object" &&
(args[0] as Record<string, unknown>).type === "resume-thread",
);
expect(resumeCalls).toHaveLength(1);
expect(resumeCalls[0][0]).toMatchObject({
@@ -286,7 +287,7 @@ describe("WorkflowManager — crash recovery (Phase 3)", () => {
});
const appendCalls = logStore.append.mock.calls.filter(
([entry]: [{ type: string }]) => entry.type === "thread_command_event",
(args: any[]) => (args[0] as { type: string }).type === "thread_command_event",
);
expect(appendCalls).toHaveLength(1);
expect(appendCalls[0][0]).toMatchObject({
@@ -313,7 +314,7 @@ describe("WorkflowManager — crash recovery (Phase 3)", () => {
mgr.startWorkflow("my-wf", payload);
const startedCall = logStore.upsertWorkflowRun.mock.calls.find(
([entry]: [{ type: string }]) => entry.type === "started",
(args: any[]) => (args[0] as { type: string }).type === "started",
);
expect(startedCall).toBeDefined();
const logEntry = startedCall?.[0] as { payload: string | null };
@@ -79,6 +79,7 @@ function makeLogStore() {
getThreadEvents: vi.fn(() => []),
archiveLogs: vi.fn(() => ({ days: [], vacuumed: false })),
close: vi.fn(),
getAllWorkflowRuns: vi.fn(() => []),
};
}
@@ -126,7 +127,7 @@ describe("WorkflowManager — drainAndRespawn (Phase 3 hot reload)", () => {
await drainPromise;
const interruptedCalls = logStore.upsertWorkflowRun.mock.calls.filter(
([entry]: [{ type: string }]) => entry.type === "interrupted",
(args: any[]) => (args[0] as { type: string }).type === "interrupted",
);
expect(interruptedCalls).toHaveLength(2);
@@ -190,10 +191,10 @@ describe("WorkflowManager — drainAndRespawn (Phase 3 hot reload)", () => {
const newChild = mockChildren[1];
const resumeCalls = (newChild.send as ReturnType<typeof vi.fn>).mock.calls.filter(
([msg]: [unknown]) =>
msg !== null &&
typeof msg === "object" &&
(msg as Record<string, unknown>).type === "resume-thread",
(args: any[]) =>
args[0] !== null &&
typeof args[0] === "object" &&
(args[0] as Record<string, unknown>).type === "resume-thread",
);
expect(resumeCalls).toHaveLength(0);
@@ -218,10 +219,10 @@ describe("WorkflowManager — drainAndRespawn (Phase 3 hot reload)", () => {
const newChild = mockChildren[1];
const startCalls = (newChild.send as ReturnType<typeof vi.fn>).mock.calls.filter(
([msg]: [unknown]) =>
msg !== null &&
typeof msg === "object" &&
(msg as Record<string, unknown>).type === "start-thread",
(args: any[]) =>
args[0] !== null &&
typeof args[0] === "object" &&
(args[0] as Record<string, unknown>).type === "start-thread",
);
expect(startCalls).toHaveLength(1);
@@ -266,7 +267,7 @@ describe("Kernel — workflow hot reload via file-watcher (Phase 3)", () => {
// Kernel's handleWorkflowFileChange should log a workflow_reload event
// We test this via the kernel itself
const appendCalls = logStore.append.mock.calls;
const startCall = appendCalls.find(([e]: [{ type: string }]) => e.type === "start");
const startCall = appendCalls.find((args: any[]) => (args[0] as { type: string }).type === "start");
expect(startCall).toBeDefined();
const stopPromise = kernel.stop();
@@ -2,7 +2,7 @@
* Unit tests for kernel.triggerSense() — IPC issue #36.
*
* These tests use a mock child_process and a mock LogStore so they do NOT
* require better-sqlite3 to be present in the test environment.
* require a real LogStore (node:sqlite) in integration tests.
*/
import { EventEmitter } from "node:events";
@@ -58,7 +58,7 @@ vi.mock("node:child_process", () => ({
const { createKernel } = await import("../kernel.js");
// ---------------------------------------------------------------------------
// Mock LogStore factory (avoids better-sqlite3 dependency)
// Mock LogStore factory (avoids SQLite I/O in this unit test)
// ---------------------------------------------------------------------------
function makeMockLogStore() {
@@ -78,6 +78,7 @@ function makeLogStore() {
appendWithWorkflowUpdate: vi.fn(),
getWorkflowRun: vi.fn(() => null),
getActiveWorkflowRuns: vi.fn(() => []),
getAllWorkflowRuns: vi.fn(() => []),
getTriggerPayload: vi.fn(() => null),
getThreadEvents: vi.fn(() => []),
archiveLogs: vi.fn(() => ({ days: [], vacuumed: false })),
@@ -137,10 +138,10 @@ describe("kernel + workflowManager integration", () => {
// We need to check that a start-thread message was sent to the workflow worker
const workflowWorker = mockChildren.find((c) =>
(c.send as ReturnType<typeof vi.fn>).mock.calls.some(
([msg]: [unknown]) =>
msg !== null &&
typeof msg === "object" &&
(msg as Record<string, unknown>).type === "start-thread",
(args: unknown[]) =>
args[0] !== null &&
typeof args[0] === "object" &&
(args[0] as Record<string, unknown>).type === "start-thread",
),
);
expect(workflowWorker).toBeDefined();
@@ -212,10 +213,10 @@ describe("kernel + workflowManager integration", () => {
// No workflow worker should have been spawned (only the sense group worker)
const workflowWorkerSpawned = mockChildren.some((c) =>
(c.send as ReturnType<typeof vi.fn>).mock.calls.some(
([msg]: [unknown]) =>
msg !== null &&
typeof msg === "object" &&
(msg as Record<string, unknown>).type === "start-thread",
(args: unknown[]) =>
args[0] !== null &&
typeof args[0] === "object" &&
(args[0] as Record<string, unknown>).type === "start-thread",
),
);
expect(workflowWorkerSpawned).toBe(false);
@@ -2,15 +2,15 @@ import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import Database from "better-sqlite3";
import { drizzle } from "drizzle-orm/better-sqlite3";
import { DatabaseSync } from "node:sqlite";
import { drizzle } from "drizzle-orm/node-sqlite";
import { integer, real, sqliteTable } from "drizzle-orm/sqlite-core";
import { describe, expect, it } from "vitest";
import { createBlobStore } from "../blob-store.js";
import { parseParentMessage } from "../ipc.js";
import { executeCompute, openPeerDb, openSenseDb, runMigrations } from "../sense-runtime.js";
import type { DrizzleDB, PeerMap, SenseRuntime } from "../sense-runtime.js";
import type { ComputeFn, DrizzleDB, PeerMap, SenseRuntime } from "../sense-runtime.js";
// ---------------------------------------------------------------------------
// Helpers
@@ -49,7 +49,7 @@ const samples = sqliteTable("samples", {
describe("runMigrations", () => {
it("creates table via SQL migration file", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const migrationsDir = makeTempMigrationsDir(INIT_SQL);
const result = runMigrations(sqlite, migrationsDir);
@@ -64,7 +64,7 @@ describe("runMigrations", () => {
});
it("runs multiple migrations in lexicographic order", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const dir = mkdtempSync(join(tmpdir(), "nerve-multi-"));
writeFileSync(join(dir, "0001_init.sql"), INIT_SQL);
@@ -81,7 +81,7 @@ describe("runMigrations", () => {
});
it("returns ok when migrations directory is empty", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const dir = makeTempMigrationsDirEmpty();
const result = runMigrations(sqlite, dir);
expect(result.ok).toBe(true);
@@ -89,14 +89,14 @@ describe("runMigrations", () => {
});
it("returns err when migrations directory does not exist", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const result = runMigrations(sqlite, "/nonexistent/path/migrations");
expect(result.ok).toBe(false);
sqlite.close();
});
it("returns err when a migration SQL is invalid", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const dir = mkdtempSync(join(tmpdir(), "nerve-bad-sql-"));
writeFileSync(join(dir, "0001_bad.sql"), "NOT VALID SQL !!!;");
const result = runMigrations(sqlite, dir);
@@ -141,7 +141,7 @@ describe("openPeerDb", () => {
it("opens an existing db in read-only mode", () => {
// Create a writable db first
const dbPath = makeTempDbPath();
const sqlite = new Database(dbPath);
const sqlite = new DatabaseSync(dbPath);
sqlite.exec(INIT_SQL);
sqlite.prepare("INSERT INTO samples (ts, value) VALUES (1, 42.0)").run();
sqlite.close();
@@ -168,13 +168,13 @@ describe("openPeerDb", () => {
// ---------------------------------------------------------------------------
describe("executeCompute", () => {
function makeRuntime(computeFn: (db: DrizzleDB, peers: PeerMap) => Promise<unknown | null>): {
function makeRuntime(computeFn: ComputeFn): {
runtime: SenseRuntime;
sqlite: Database.Database;
sqlite: DatabaseSync;
} {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
sqlite.exec(INIT_SQL);
const db = drizzle(sqlite) as DrizzleDB;
const db = drizzle({ client: sqlite }) as DrizzleDB;
return {
runtime: { name: "test-sense", db, compute: computeFn },
sqlite,
@@ -226,10 +226,10 @@ describe("executeCompute", () => {
it("compute can read from peers", async () => {
// Set up a peer db with data
const peerSqlite = new Database(":memory:");
const peerSqlite = new DatabaseSync(":memory:");
peerSqlite.exec(INIT_SQL);
peerSqlite.prepare("INSERT INTO samples (ts, value) VALUES (100, 3.14)").run();
const peerDb = drizzle(peerSqlite) as DrizzleDB;
const peerDb = drizzle({ client: peerSqlite }) as DrizzleDB;
const peers: PeerMap = { "other-sense": peerDb };
@@ -248,9 +248,9 @@ describe("executeCompute", () => {
});
it("write to own db does not affect peer db (isolation)", async () => {
const peerSqlite = new Database(":memory:");
const peerSqlite = new DatabaseSync(":memory:");
peerSqlite.exec(INIT_SQL);
const peerDb = drizzle(peerSqlite) as DrizzleDB;
const peerDb = drizzle({ client: peerSqlite }) as DrizzleDB;
const peers: PeerMap = { "peer-sense": peerDb };
const { runtime, sqlite } = makeRuntime(async (db) => {
@@ -403,7 +403,7 @@ describe("parseParentMessage", () => {
describe("runMigrations journal", () => {
it("does not re-run an already-applied migration", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const dir = mkdtempSync(join(tmpdir(), "nerve-journal-"));
writeFileSync(join(dir, "0001_init.sql"), INIT_SQL);
@@ -430,7 +430,7 @@ describe("runMigrations journal", () => {
});
it("tracks migrations in _migrations table", () => {
const sqlite = new Database(":memory:");
const sqlite = new DatabaseSync(":memory:");
const dir = mkdtempSync(join(tmpdir(), "nerve-journal2-"));
writeFileSync(join(dir, "0001_init.sql"), INIT_SQL);
@@ -76,6 +76,7 @@ function makeLogStore() {
getThreadEvents: vi.fn(() => []),
archiveLogs: vi.fn(() => ({ days: [], vacuumed: false })),
close: vi.fn(),
getAllWorkflowRuns: vi.fn(() => []),
};
}
+34 -17
View File
@@ -9,8 +9,7 @@
import { mkdirSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import Database from "better-sqlite3";
import type BetterSqlite3 from "better-sqlite3";
import { DatabaseSync, type StatementSync } from "node:sqlite";
import {
DEFAULT_LOG_RETENTION_MS,
@@ -184,7 +183,23 @@ function buildJsonlBody(rows: SqlLogRow[]): string {
return `${lines.join("\n")}\n`;
}
function runOptionalVacuum(sqlite: BetterSqlite3.Database, vacuum?: boolean): boolean {
function runInTransaction<T>(db: DatabaseSync, fn: () => T): T {
db.exec("BEGIN IMMEDIATE");
try {
const out = fn();
db.exec("COMMIT");
return out;
} catch (e) {
try {
db.exec("ROLLBACK");
} catch {
// ignore rollback errors
}
throw e;
}
}
function runOptionalVacuum(sqlite: DatabaseSync, vacuum?: boolean): boolean {
if (vacuum !== true) return false;
sqlite.exec("VACUUM");
return true;
@@ -199,7 +214,7 @@ function resolveArchiveStartDay(watermark: string | null, minDay: string): strin
function runArchiveDayLoop(
dbPath: string,
options: ArchiveLogsOptions,
selectLogsForDayStmt: BetterSqlite3.Statement,
selectLogsForDayStmt: StatementSync,
archiveDayTx: (day: string, start: number, endExclusive: number) => void,
startDay: string,
lastDay: string,
@@ -235,8 +250,8 @@ function runArchiveDayLoop(
export function createLogStore(dbPath: string): LogStore {
mkdirSync(dirname(dbPath), { recursive: true });
const sqlite: BetterSqlite3.Database = new Database(dbPath);
sqlite.pragma("journal_mode = WAL");
const sqlite = new DatabaseSync(dbPath);
sqlite.exec("PRAGMA journal_mode=WAL");
sqlite.exec(SCHEMA_SQL);
const insertStmt = sqlite.prepare(
@@ -288,8 +303,8 @@ export function createLogStore(dbPath: string): LogStore {
"DELETE FROM logs WHERE ts >= @start AND ts < @endExclusive",
);
const upsertWorkflowRunTx = sqlite.transaction(
(entry: Omit<LogEntry, "id">, run: WorkflowRun) => {
function upsertWorkflowRunTx(entry: Omit<LogEntry, "id">, run: WorkflowRun): LogEntry {
return runInTransaction(sqlite, () => {
const info = insertStmt.run({
source: entry.source,
type: entry.type,
@@ -304,8 +319,8 @@ export function createLogStore(dbPath: string): LogStore {
ts: run.ts,
});
return { ...entry, id: Number(info.lastInsertRowid) };
},
);
});
}
function append(entry: Omit<LogEntry, "id">): LogEntry {
const info = insertStmt.run({
@@ -320,7 +335,7 @@ export function createLogStore(dbPath: string): LogStore {
function query(filter: LogQuery = {}): LogEntry[] {
const conditions: string[] = [];
const params: Record<string, unknown> = {};
const params: Record<string, string | number> = {};
if (filter.source !== undefined) {
conditions.push("source = @source");
@@ -376,11 +391,11 @@ export function createLogStore(dbPath: string): LogStore {
}
function upsertWorkflowRun(entry: Omit<LogEntry, "id">, run: WorkflowRun): LogEntry {
return upsertWorkflowRunTx(entry, run) as LogEntry;
return upsertWorkflowRunTx(entry, run);
}
function appendWithWorkflowUpdate(entry: Omit<LogEntry, "id">, run: WorkflowRun): LogEntry {
return upsertWorkflowRunTx(entry, run) as LogEntry;
return upsertWorkflowRunTx(entry, run);
}
function getWorkflowRun(runId: string): WorkflowRun | null {
@@ -460,10 +475,12 @@ export function createLogStore(dbPath: string): LogStore {
return result;
}
const archiveDayTx = sqlite.transaction((day: string, start: number, endExclusive: number) => {
deleteLogsForDayStmt.run({ start, endExclusive });
setMetaStmt.run({ key: LOG_ARCHIVE_META_KEY, value: day });
});
function archiveDayTx(day: string, start: number, endExclusive: number): void {
runInTransaction(sqlite, () => {
deleteLogsForDayStmt.run({ start, endExclusive });
setMetaStmt.run({ key: LOG_ARCHIVE_META_KEY, value: day });
});
}
function readWatermark(): string | null {
const raw = getMeta(LOG_ARCHIVE_META_KEY);
+25 -25
View File
@@ -1,9 +1,9 @@
import { mkdirSync, readFileSync, readdirSync } from "node:fs";
import { dirname, join } from "node:path";
import { DatabaseSync } from "node:sqlite";
import Database from "better-sqlite3";
import { drizzle } from "drizzle-orm/better-sqlite3";
import type { BetterSQLite3Database } from "drizzle-orm/better-sqlite3";
import { drizzle } from "drizzle-orm/node-sqlite";
import type { NodeSQLiteDatabase } from "drizzle-orm/node-sqlite";
import type { Result } from "@uncaged/nerve-core";
import { err, ok } from "@uncaged/nerve-core";
@@ -11,7 +11,7 @@ import { err, ok } from "@uncaged/nerve-core";
import type { BlobStore } from "./blob-store.js";
/** A Drizzle DB instance (schema-generic) */
export type DrizzleDB = BetterSQLite3Database<Record<string, never>>;
export type DrizzleDB = NodeSQLiteDatabase<Record<string, never>>;
/** Read-only map of peer sense name → their Drizzle DB */
export type PeerMap = Readonly<Record<string, DrizzleDB>>;
@@ -42,7 +42,7 @@ export type SenseRuntime = {
compute: ComputeFn;
};
function ensureMigrationsTable(sqlite: Database.Database): Result<void> {
function ensureMigrationsTable(sqlite: DatabaseSync): Result<void> {
try {
sqlite.exec(
`CREATE TABLE IF NOT EXISTS _migrations (
@@ -69,11 +69,7 @@ function listMigrationFiles(migrationsDir: string): Result<string[]> {
}
}
function applyMigrationFile(
sqlite: Database.Database,
file: string,
filePath: string,
): Result<void> {
function applyMigrationFile(sqlite: DatabaseSync, file: string, filePath: string): Result<void> {
let sql: string;
try {
sql = readFileSync(filePath, "utf8");
@@ -83,13 +79,18 @@ function applyMigrationFile(
}
const insertJournal = sqlite.prepare("INSERT INTO _migrations (name, applied_at) VALUES (?, ?)");
sqlite.exec("BEGIN IMMEDIATE");
try {
sqlite.transaction(() => {
sqlite.exec(sql);
insertJournal.run(file, Date.now());
})();
sqlite.exec(sql);
insertJournal.run(file, Date.now());
sqlite.exec("COMMIT");
return ok(undefined);
} catch (e) {
try {
sqlite.exec("ROLLBACK");
} catch {
// ignore secondary errors during rollback
}
const msg = e instanceof Error ? e.message : String(e);
return err(new Error(`Migration "${file}" failed: ${msg}`));
}
@@ -97,10 +98,10 @@ function applyMigrationFile(
/**
* Run all *.sql migration files in the given directory against a
* better-sqlite3 Database, in lexicographic order.
* `node:sqlite` DatabaseSync, in lexicographic order.
* Tracks applied migrations in _migrations table to avoid re-running.
*/
export function runMigrations(sqlite: Database.Database, migrationsDir: string): Result<void> {
export function runMigrations(sqlite: DatabaseSync, migrationsDir: string): Result<void> {
const tableResult = ensureMigrationsTable(sqlite);
if (!tableResult.ok) return tableResult;
@@ -129,14 +130,13 @@ export function runMigrations(sqlite: Database.Database, migrationsDir: string):
export function openSenseDb(
dbPath: string,
migrationsDir: string,
): Result<{ sqlite: Database.Database; db: DrizzleDB }> {
let sqlite: Database.Database;
): Result<{ sqlite: DatabaseSync; db: DrizzleDB }> {
let sqlite: DatabaseSync;
try {
mkdirSync(dirname(dbPath), { recursive: true });
sqlite = new Database(dbPath);
// WAL mode for better concurrent read performance
sqlite.pragma("journal_mode = WAL");
sqlite = new DatabaseSync(dbPath);
sqlite.exec("PRAGMA journal_mode=WAL");
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
return err(new Error(`Failed to open database "${dbPath}": ${msg}`));
@@ -145,7 +145,7 @@ export function openSenseDb(
const migResult = runMigrations(sqlite, migrationsDir);
if (!migResult.ok) return migResult;
const db = drizzle(sqlite) as DrizzleDB;
const db = drizzle({ client: sqlite }) as DrizzleDB;
return ok({ sqlite, db });
}
@@ -153,16 +153,16 @@ export function openSenseDb(
* Open a peer sense DB in read-only mode (no migrations).
*/
export function openPeerDb(dbPath: string): Result<DrizzleDB> {
let sqlite: Database.Database;
let sqlite: DatabaseSync;
try {
sqlite = new Database(dbPath, { readonly: true });
sqlite = new DatabaseSync(dbPath, { readOnly: true });
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
return err(new Error(`Failed to open peer database "${dbPath}" (readonly): ${msg}`));
}
return ok(drizzle(sqlite) as DrizzleDB);
return ok(drizzle({ client: sqlite }) as DrizzleDB);
}
/**
-8
View File
@@ -1,8 +0,0 @@
import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts", "src/sense-worker.ts"],
format: ["esm"],
dts: true,
clean: true,
});
+1032 -492
View File
File diff suppressed because it is too large Load Diff
-1
View File
@@ -3,5 +3,4 @@ packages:
onlyBuiltDependencies:
- "@biomejs/biome"
- better-sqlite3
- esbuild