nerve/.knowledge/storage-layer.md

Storage Layer

Nerve uses multiple storage systems designed for different data types and access patterns.

Core Storage Components

1. Log Store (logs.db)

Append-only audit trail implemented in SQLite with WAL mode.

Schema:

• logs — all system events (signals, workflow transitions, sense outputs)
• meta — key-value store for system metadata
• workflow_runs — materialized view of workflow execution state

Key Features:

• Atomic workflow state updates via transactions
• Thread message persistence for crash recovery
• Configurable log archival to JSONL files
• Full-text search across log entries

2. Sense Databases

Each sense group gets its own SQLite database for private state.

Characteristics:

• Isolated per sense group (e.g., system-senses.db)
• Managed by individual sense compute functions
• Drizzle ORM integration for schema management
• No cross-sense data sharing

3. Knowledge Store (knowledge.db)

Vector-enabled search index for project context.

Contents:

• Chunked source files with embeddings
• Curated knowledge cards from .knowledge/
• Semantic search capabilities
• Global vs. repo-scoped search modes

4. Blob Store (CAS)

Content-addressable storage for large artifacts.

Design:

• SHA-256 based file naming
• Automatic deduplication
• Used for workflow artifacts and large payloads

Consistency & Isolation Mechanisms

SQLite WAL Mode

All SQLite databases use PRAGMA journal_mode=WAL for:

• Writer-reader concurrency — readers don't block writers
• Atomic writes — each transaction is fully applied or rolled back
• Crash recovery — WAL provides consistent state after crashes

Transaction Management

Log Store Transactions

Uses BEGIN IMMEDIATE transactions (packages/store/src/log-store.ts):

function runInTransaction<T>(db: DatabaseSync, fn: () => T): T {
  db.exec("BEGIN IMMEDIATE");  // Exclusive write lock
  try {
    const result = fn();
    db.exec("COMMIT");
    return result;
  } catch (e) {
    db.exec("ROLLBACK");
    throw e;
  }
}

Key Operations:

• upsertWorkflowRun() — atomically writes log entry + workflow state
• archiveLogs() — transactional export + delete + watermark update

Sense Database Isolation

• Each sense group has its own SQLite file (e.g., system-senses.db)
• No cross-sense transactions or coordination required
• Independent schema migrations per sense
• Private _signals table for signal history retention

Process-Level Isolation

Worker Process Architecture

• One worker per sense group — prevents data races within group
• One worker per workflow type — isolated execution contexts
• No shared memory — all communication via IPC messages

Concurrency Control

Workflow manager enforces limits per workflow:

workflows:
  my-workflow:
    concurrency: 2        # Max parallel threads
    overflow: "queue"     # or "drop" 
    maxQueue: 10         # Queue depth limit

Consistency Guarantees & Failure Modes

Strong Consistency (Single Database):

1. Within Log Store — ACID transactions with immediate consistency
2. Within Sense DB — WAL mode ensures atomic commits per database
3. Workflow State — upsertWorkflowRun() atomically updates log + materialized view

No Cross-Database Consistency:

• No distributed transactions across multiple SQLite files
• Log Store and Sense Databases can temporarily diverge during failures
• Signal emission and workflow triggering are separate, non-atomic operations

Failure Recovery Mechanisms:

• Sense worker crash: State rebuilt from sense SQLite database on respawn
• Workflow worker crash: Thread state recovered from log store message history
• Kernel crash: All workers respawned, state recovered from persistent stores
• Log Store corruption: WAL recovery on database open
• Sense DB corruption: Migrations re-run, _signals table rebuilt if needed

Rollback Scenarios:

• Log write failure: Transaction rolled back, no state changes persisted
• Sense compute failure: Error logged, no signal/workflow emitted
• Workflow failure: Thread marked as failed in materialized view
• IPC failure: Worker respawned, pending operations lost (not rolled back)

Archive Strategy

Logs older than retention window (default 30 days) are:

1. Exported to data/archive/logs/YYYY-MM-DD.jsonl
2. Deleted from active database
3. Watermark updated to prevent re-processing

This keeps the active database size bounded while preserving audit trails.