xiaoju
9c832b0e21
7 cards updated, 4 new cards added. Topics: signal-routing, worker-isolation, storage-layer, adapter-isolation, sense contracts, workflow runtime enforcement, coding conventions details. 小橘 <xiaoju@shazhou.work>
132 lines
4.5 KiB
Markdown
132 lines
4.5 KiB
Markdown
# Storage Layer
|
|
|
|
Nerve uses multiple storage systems designed for different data types and access patterns.
|
|
|
|
## Core Storage Components
|
|
|
|
### 1. Log Store (`logs.db`)
|
|
Append-only audit trail implemented in SQLite with WAL mode.
|
|
|
|
**Schema:**
|
|
- `logs` — all system events (signals, workflow transitions, sense outputs)
|
|
- `meta` — key-value store for system metadata
|
|
- `workflow_runs` — materialized view of workflow execution state
|
|
|
|
**Key Features:**
|
|
- Atomic workflow state updates via transactions
|
|
- Thread message persistence for crash recovery
|
|
- Configurable log archival to JSONL files
|
|
- Full-text search across log entries
|
|
|
|
### 2. Sense Databases
|
|
Each sense group gets its own SQLite database for private state.
|
|
|
|
**Characteristics:**
|
|
- Isolated per sense group (e.g., `system-senses.db`)
|
|
- Managed by individual sense compute functions
|
|
- Drizzle ORM integration for schema management
|
|
- No cross-sense data sharing
|
|
|
|
### 3. Knowledge Store (`knowledge.db`)
|
|
Vector-enabled search index for project context.
|
|
|
|
**Contents:**
|
|
- Chunked source files with embeddings
|
|
- Curated knowledge cards from `.knowledge/`
|
|
- Semantic search capabilities
|
|
- Global vs. repo-scoped search modes
|
|
|
|
### 4. Blob Store (CAS)
|
|
Content-addressable storage for large artifacts.
|
|
|
|
**Design:**
|
|
- SHA-256 based file naming
|
|
- Automatic deduplication
|
|
- Used for workflow artifacts and large payloads
|
|
|
|
## Consistency & Isolation Mechanisms
|
|
|
|
### SQLite WAL Mode
|
|
All SQLite databases use `PRAGMA journal_mode=WAL` for:
|
|
- **Writer-reader concurrency** — readers don't block writers
|
|
- **Atomic writes** — each transaction is fully applied or rolled back
|
|
- **Crash recovery** — WAL provides consistent state after crashes
|
|
|
|
### Transaction Management
|
|
|
|
#### Log Store Transactions
|
|
Uses `BEGIN IMMEDIATE` transactions (`packages/store/src/log-store.ts`):
|
|
```typescript
|
|
function runInTransaction<T>(db: DatabaseSync, fn: () => T): T {
|
|
db.exec("BEGIN IMMEDIATE"); // Exclusive write lock
|
|
try {
|
|
const result = fn();
|
|
db.exec("COMMIT");
|
|
return result;
|
|
} catch (e) {
|
|
db.exec("ROLLBACK");
|
|
throw e;
|
|
}
|
|
}
|
|
```
|
|
|
|
**Key Operations:**
|
|
- `upsertWorkflowRun()` — atomically writes log entry + workflow state
|
|
- `archiveLogs()` — transactional export + delete + watermark update
|
|
|
|
#### Sense Database Isolation
|
|
- Each sense group has its own SQLite file (e.g., `system-senses.db`)
|
|
- No cross-sense transactions or coordination required
|
|
- Independent schema migrations per sense
|
|
- Private `_signals` table for signal history retention
|
|
|
|
### Process-Level Isolation
|
|
|
|
#### Worker Process Architecture
|
|
- **One worker per sense group** — prevents data races within group
|
|
- **One worker per workflow type** — isolated execution contexts
|
|
- **No shared memory** — all communication via IPC messages
|
|
|
|
#### Concurrency Control
|
|
Workflow manager enforces limits per workflow:
|
|
```yaml
|
|
workflows:
|
|
my-workflow:
|
|
concurrency: 2 # Max parallel threads
|
|
overflow: "queue" # or "drop"
|
|
maxQueue: 10 # Queue depth limit
|
|
```
|
|
|
|
### Consistency Guarantees & Failure Modes
|
|
|
|
**Strong Consistency (Single Database)**:
|
|
1. **Within Log Store** — ACID transactions with immediate consistency
|
|
2. **Within Sense DB** — WAL mode ensures atomic commits per database
|
|
3. **Workflow State** — `upsertWorkflowRun()` atomically updates log + materialized view
|
|
|
|
**No Cross-Database Consistency**:
|
|
- No distributed transactions across multiple SQLite files
|
|
- Log Store and Sense Databases can temporarily diverge during failures
|
|
- Signal emission and workflow triggering are separate, non-atomic operations
|
|
|
|
**Failure Recovery Mechanisms**:
|
|
- **Sense worker crash**: State rebuilt from sense SQLite database on respawn
|
|
- **Workflow worker crash**: Thread state recovered from log store message history
|
|
- **Kernel crash**: All workers respawned, state recovered from persistent stores
|
|
- **Log Store corruption**: WAL recovery on database open
|
|
- **Sense DB corruption**: Migrations re-run, `_signals` table rebuilt if needed
|
|
|
|
**Rollback Scenarios**:
|
|
- **Log write failure**: Transaction rolled back, no state changes persisted
|
|
- **Sense compute failure**: Error logged, no signal/workflow emitted
|
|
- **Workflow failure**: Thread marked as failed in materialized view
|
|
- **IPC failure**: Worker respawned, pending operations lost (not rolled back)
|
|
|
|
## Archive Strategy
|
|
|
|
Logs older than retention window (default 30 days) are:
|
|
1. Exported to `data/archive/logs/YYYY-MM-DD.jsonl`
|
|
2. Deleted from active database
|
|
3. Watermark updated to prevent re-processing
|
|
|
|
This keeps the active database size bounded while preserving audit trails. |