# Storage Layer

Nerve uses multiple storage systems designed for different data types and access patterns.

## Core Storage Components

### 1. Log Store (`logs.db`)
Append-only audit trail implemented in SQLite with WAL mode.

**Schema:**
- `logs` — all system events (signals, workflow transitions, sense outputs)
- `meta` — key-value store for system metadata
- `workflow_runs` — materialized view of workflow execution state

**Key Features:**
- Atomic workflow state updates via transactions
- Thread message persistence for crash recovery
- Configurable log archival to JSONL files
- Full-text search across log entries

### 2. Sense Databases
Each sense group gets its own SQLite database for private state.

**Characteristics:**
- Isolated per sense group (e.g., `system-senses.db`)
- Managed by individual sense compute functions
- Drizzle ORM integration for schema management
- No cross-sense data sharing

### 3. Knowledge Store (`knowledge.db`)
Vector-enabled search index for project context.

**Contents:**
- Chunked source files with embeddings
- Curated knowledge cards from `.knowledge/`
- Semantic search capabilities
- Global vs. repo-scoped search modes

### 4. Blob Store (CAS)
Content-addressable storage for large artifacts.

**Design:**
- SHA-256 based file naming
- Automatic deduplication
- Used for workflow artifacts and large payloads

## Consistency & Isolation Mechanisms

### SQLite WAL Mode
All SQLite databases use `PRAGMA journal_mode=WAL` for:
- **Writer-reader concurrency** — readers don't block writers
- **Atomic writes** — each transaction is fully applied or rolled back
- **Crash recovery** — WAL provides consistent state after crashes

### Transaction Management

#### Log Store Transactions
Uses `BEGIN IMMEDIATE` transactions (`packages/store/src/log-store.ts`):
```typescript
function runInTransaction<T>(db: DatabaseSync, fn: () => T): T {
  db.exec("BEGIN IMMEDIATE");  // Exclusive write lock
  try {
    const result = fn();
    db.exec("COMMIT");
    return result;
  } catch (e) {
    db.exec("ROLLBACK");
    throw e;
  }
}
```

**Key Operations:**
- `upsertWorkflowRun()` — atomically writes log entry + workflow state
- `archiveLogs()` — transactional export + delete + watermark update

#### Sense Database Isolation
- Each sense group has its own SQLite file (e.g., `system-senses.db`)
- No cross-sense transactions or coordination required
- Independent schema migrations per sense
- Private `_signals` table for signal history retention

### Process-Level Isolation

#### Worker Process Architecture
- **One worker per sense group** — prevents data races within group
- **One worker per workflow type** — isolated execution contexts  
- **No shared memory** — all communication via IPC messages

#### Concurrency Control
Workflow manager enforces limits per workflow:
```yaml
workflows:
  my-workflow:
    concurrency: 2        # Max parallel threads
    overflow: "queue"     # or "drop" 
    maxQueue: 10         # Queue depth limit
```

### Consistency Guarantees & Failure Modes

**Strong Consistency (Single Database)**:
1. **Within Log Store** — ACID transactions with immediate consistency
2. **Within Sense DB** — WAL mode ensures atomic commits per database
3. **Workflow State** — `upsertWorkflowRun()` atomically updates log + materialized view

**No Cross-Database Consistency**:
- No distributed transactions across multiple SQLite files
- Log Store and Sense Databases can temporarily diverge during failures
- Signal emission and workflow triggering are separate, non-atomic operations

**Failure Recovery Mechanisms**:
- **Sense worker crash**: State rebuilt from sense SQLite database on respawn
- **Workflow worker crash**: Thread state recovered from log store message history
- **Kernel crash**: All workers respawned, state recovered from persistent stores
- **Log Store corruption**: WAL recovery on database open
- **Sense DB corruption**: Migrations re-run, `_signals` table rebuilt if needed

**Rollback Scenarios**:
- **Log write failure**: Transaction rolled back, no state changes persisted
- **Sense compute failure**: Error logged, no signal/workflow emitted
- **Workflow failure**: Thread marked as failed in materialized view
- **IPC failure**: Worker respawned, pending operations lost (not rolled back)

## Archive Strategy

Logs older than retention window (default 30 days) are:
1. Exported to `data/archive/logs/YYYY-MM-DD.jsonl`
2. Deleted from active database
3. Watermark updated to prevent re-processing

This keeps the active database size bounded while preserving audit trails.