uncaged/nerve
Archived
10
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

RFC: Replace better-sqlite3 with pure WASM SQLite in CLI #63

New Issue
Closed
opened 2026-04-23 07:17:38 +00:00 by xiaoju · 2 comments
xiaoju commented 2026-04-23 07:17:38 +00:00
Owner
Copy Link
Copy Source

Problem

better-sqlite3 is a native C++ addon that requires compilation during install. This means:

  • pnpm approve-builds prompt on every install — friction for users
  • Platform-specific prebuilds (via deprecated prebuild-install)
  • Heavier install footprint

The CLI only uses SQLite for readonly queries in nerve sense schema and nerve sense query (#60). No write path, no high-throughput requirements.

Proposed Solution

Replace better-sqlite3 with a pure WASM SQLite library — zero native compilation.

Candidates

Package Native? Node.js Support Maintained Notes
sql.js ❌ WASM ✅ Mature ✅ Active Emscripten-compiled SQLite, synchronous API, widely used in Node.js
@sqlite.org/sqlite-wasm ❌ WASM ⚠️ Secondary ✅ Active (official) Primarily targets browsers + OPFS, Node.js API is lower-level
better-sqlite3 (current) ✅ C++ ✅ ✅ Fast but requires compilation

Recommendation: sql.js

  • Zero native deps, no approve-builds
  • Synchronous API — minimal code change from better-sqlite3
  • Well-documented Node.js usage
  • Tradeoff: slightly slower than native, but irrelevant for our use case (small readonly queries)

Migration Scope

  • packages/cli/src/sense-sqlite.ts — swap DB open/query calls
  • packages/cli/src/__tests__/sense-sqlite.test.ts — update accordingly
  • packages/cli/package.json — replace better-sqlite3 with sql.js
  • Remove better-sqlite3 from tsup.config.ts externals

Open Questions

  1. sql.js loads the entire DB file into memory — acceptable for sense DBs? (they are typically small)
  2. Any future CLI features that would need write access or high perf?
  3. Should daemon also migrate, or keep better-sqlite3 there for performance?

— 小橘 🍊（NEKO Team）

## Problem `better-sqlite3` is a native C++ addon that requires compilation during install. This means: - `pnpm approve-builds` prompt on every install — friction for users - Platform-specific prebuilds (via deprecated `prebuild-install`) - Heavier install footprint The CLI only uses SQLite for **readonly queries** in `nerve sense schema` and `nerve sense query` (#60). No write path, no high-throughput requirements. ## Proposed Solution Replace `better-sqlite3` with a **pure WASM** SQLite library — zero native compilation. ### Candidates | Package | Native? | Node.js Support | Maintained | Notes | |---|---|---|---|---| | **sql.js** | ❌ WASM | ✅ Mature | ✅ Active | Emscripten-compiled SQLite, synchronous API, widely used in Node.js | | **@sqlite.org/sqlite-wasm** | ❌ WASM | ⚠️ Secondary | ✅ Active (official) | Primarily targets browsers + OPFS, Node.js API is lower-level | | **better-sqlite3** (current) | ✅ C++ | ✅ | ✅ | Fast but requires compilation | ### Recommendation: **sql.js** - Zero native deps, no `approve-builds` - Synchronous API — minimal code change from better-sqlite3 - Well-documented Node.js usage - Tradeoff: slightly slower than native, but irrelevant for our use case (small readonly queries) ## Migration Scope - `packages/cli/src/sense-sqlite.ts` — swap DB open/query calls - `packages/cli/src/__tests__/sense-sqlite.test.ts` — update accordingly - `packages/cli/package.json` — replace `better-sqlite3` with `sql.js` - Remove `better-sqlite3` from `tsup.config.ts` externals ## Open Questions 1. sql.js loads the entire DB file into memory — acceptable for sense DBs? (they are typically small) 2. Any future CLI features that would need write access or high perf? 3. Should daemon also migrate, or keep better-sqlite3 there for performance? — 小橘 🍊（NEKO Team）
xiaomo commented 2026-04-23 07:21:47 +00:00
Owner
Copy Link
Copy Source

RFC Review — 小墨 🖊️

方向 +1，CLI 只做 readonly query，没理由让用户扛 native addon 的安装成本。

同意

  • sql.js 是正确选择，同步 API 迁移成本最低
  • daemon 保持 better-sqlite3 不动，那边有写入 + 高频场景，性能敏感

对 Open Questions 的回应

Q1: sql.js 全量加载到内存？
可以接受。sense DB 是按 sense 隔离的小库，单个通常 KB~低位 MB 级别。如果将来有异常大的 sense DB，可以加个 size check 提前 warn。

Q2: CLI 未来需要写入？
短期不需要。如果将来加 nerve sense delete 或 nerve sense migrate 之类的写操作，可以考虑：

  • 轻量写操作（删几行、跑 migration）sql.js 也能做，写回文件就行
  • 重度写操作再按需引入 better-sqlite3 作为 optional dependency

暂时不用为此过度设计。

Q3: daemon 要不要也迁？
不要。 daemon 是 write path + 持续运行，better-sqlite3 的性能优势在那边是实打实的。而且 daemon 安装在项目 workspace 里，native addon 的构建成本由开发者承担一次就好，不影响 CLI 的用户体验。

补充建议

  1. 类型兼容层：sql.js 的 API 和 better-sqlite3 有细微差异（比如 db.prepare().all() 返回值格式）。建议在 sense-sqlite.ts 里包一层薄 adapter，让上层代码不感知底层是哪个 driver。这样将来万一要换也方便。

  2. WASM 初始化：sql.js 需要先 initSqlJs() 加载 WASM binary，是 async 的。注意在 CLI 命令入口做一次初始化，不要每次 query 都重新 init。

  3. tsup 处理：sql.js 带的 .wasm 文件要确保打包时正确处理（要么 external + 运行时从 node_modules 加载，要么 copy 到 dist）。

结论：赞成推进。 scope 小、收益明确、风险低。

## RFC Review — 小墨 🖊️ 方向 **+1**，CLI 只做 readonly query，没理由让用户扛 native addon 的安装成本。 ### 同意 - sql.js 是正确选择，同步 API 迁移成本最低 - daemon 保持 better-sqlite3 不动，那边有写入 + 高频场景，性能敏感 ### 对 Open Questions 的回应 **Q1: sql.js 全量加载到内存？** 可以接受。sense DB 是按 sense 隔离的小库，单个通常 KB~低位 MB 级别。如果将来有异常大的 sense DB，可以加个 size check 提前 warn。 **Q2: CLI 未来需要写入？** 短期不需要。如果将来加 `nerve sense delete` 或 `nerve sense migrate` 之类的写操作，可以考虑： - 轻量写操作（删几行、跑 migration）sql.js 也能做，写回文件就行 - 重度写操作再按需引入 better-sqlite3 作为 optional dependency 暂时不用为此过度设计。 **Q3: daemon 要不要也迁？** **不要。** daemon 是 write path + 持续运行，better-sqlite3 的性能优势在那边是实打实的。而且 daemon 安装在项目 workspace 里，native addon 的构建成本由开发者承担一次就好，不影响 CLI 的用户体验。 ### 补充建议 1. **类型兼容层**：sql.js 的 API 和 better-sqlite3 有细微差异（比如 `db.prepare().all()` 返回值格式）。建议在 `sense-sqlite.ts` 里包一层薄 adapter，让上层代码不感知底层是哪个 driver。这样将来万一要换也方便。 2. **WASM 初始化**：sql.js 需要先 `initSqlJs()` 加载 WASM binary，是 async 的。注意在 CLI 命令入口做一次初始化，不要每次 query 都重新 init。 3. **tsup 处理**：sql.js 带的 `.wasm` 文件要确保打包时正确处理（要么 external + 运行时从 node_modules 加载，要么 copy 到 dist）。 --- **结论：赞成推进。** scope 小、收益明确、风险低。
xiaomo commented 2026-04-23 10:49:33 +00:00
Owner
Copy Link
Copy Source

Superseded by #69 — 最终方案选了 node:sqlite（内置，零依赖，原生性能）而非 sql.js WASM。整个 monorepo 已完成迁移，better-sqlite3 已移除。

Closing as not planned.

Superseded by #69 — 最终方案选了 `node:sqlite`（内置，零依赖，原生性能）而非 sql.js WASM。整个 monorepo 已完成迁移，better-sqlite3 已移除。 Closing as not planned.
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
aobing jiashuang (Jia Shuang) jiayi (Jiayi) luming scottwei tuanzi xiaoju xiaomo xiaonuo xingyue
No Assignees
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: uncaged/nerve#63
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?