Testing Strategy

Testing should protect the project contracts before implementation details.

Local gates that build, clean, package, or execute packages/*/dist/ must run sequentially. They intentionally share generated output, so parallel execution can create false failures even when each command passes in isolation.

Test organization follows ADR 0007: package behavior tests stay near their packages, public contracts live with their owning package, and test-only shared helpers are extracted only after there are at least two real consumers.

Repeated validation work should become a script, shared fixture, or contract helper once it appears in more than one place. Keep exact public contracts such as the MCP tool set in one source of truth and import that source from tests, packaging validation, and runtime diagnostics instead of copying lists by hand.

Test Pyramid

1. Core unit tests.
2. Storage integration tests with temporary SQLite databases.
3. CLI contract tests.
4. MCP tool contract tests.
5. End-to-end smoke tests.

Required MVP Tests

• init creates expected directories and database;
• init is idempotent;
• remember stores valid memories;
• remember rejects obvious secrets;
• secret scanning blocks common credential formats while allowing redacted examples;
• recall returns relevant FTS results;
• list filters by scope and tag;
• update records audit event;
• history returns only the requested memory's audit events;
• forget archives by default;
• hard delete requires explicit confirmation;
• bulk forget defaults to dry-run and isolates scope/tag filters;
• export produces valid versioned format;
• import dry-run reports planned changes;
• import preflight rejects the full request before writes and handles duplicate items consistently;
• SQLite mutations roll back memory, FTS, and audit writes after injected failures;
• FTS diagnostics detect missing, orphaned, duplicate, and mismatched derived rows without exposing content;
• explicit FTS repair preserves canonical memory, event, and relation rows, creates a validated owner-only restoreable backup, rejects overlapping or unsafe paths and altered schemas, removes temporary sidecars, and restores search consistency;
• multi-item imports are atomic when a later item fails;
• normal recall does not persist query text or usage metadata unless the core caller explicitly opts in, and opt-in recall events persist only a query hash plus bounded recall metadata rather than the full query;
• doctor reports tracked memory files;
• the privacy doctor profile exposes bounded counts and stable finding codes without local paths, raw config values, memory content, or secret fragments;
• MCP doctor reports aggregate store health without exposing memory content.
• staged npm artifacts complete real installed CLI and stdio MCP lifecycle flows for capture suggestion, confirmed writes, recall hooks, the exact public tool set, and healthy memory.doctor responses.
• release-tool tests reject invalid SemVer and direct workflow input interpolation, and verify local npm credentials remain ignored.

Test Data

Use fake memory content only.

Never place real user facts, credentials, or local project secrets in fixtures.

Contract Tests

MCP and CLI should have snapshot-style contract tests for:

• input validation;
• output shape;
• error codes;
• warnings.

Snapshots should avoid timestamps and generated IDs unless normalized.

MCP protocol coverage connects an SDK Client and the Nuzo server through InMemoryTransport. It asserts exact tool discovery, registered schema defaults, representative JSON responses, and invalid-input rejection without network access or a stdio subprocess.

Artifact validation reuses the MCP server's exported tool contract so new tools do not require repeated manual updates across test files and release scripts.

CLI process coverage runs the built dist/index.js entrypoint in subprocesses and asserts stable success, operational, usage, and internal exit codes plus stack-trace-free stderr.

Migration Tests

Each migration should be tested from:

• an empty store;
• the immediately previous schema;
• a small realistic fixture.

Migration tests verify an empty database, reopen a populated store to confirm idempotency, and reject a fixture with a newer unsupported user_version. Schema version 7 also opens an explicit version 6 fixture, backfills normalized capture keys, creates the active exact-key index, and proves indexed duplicate lookup after migration. Future schema bumps should add a fixture for the immediately previous schema before changing schemaVersion.

Manual Smoke Test

Before a release:

nuzo memory init
nuzo memory remember "The user prefers local-first tools." --kind preference --tag example
nuzo memory recall "local-first"
nuzo memory list
nuzo memory export --path ./memories.memory.export.json
nuzo memory doctor

Recall-quality changes should also run:

npm run benchmark:recall

The benchmark uses synthetic public data and validates relevance, noise, scope isolation, bounded results, ranking reasons, and local latency without telemetry, network calls, or embeddings. English recall is the primary quality group and has its own strict top-1, recall, and zero-noise envelope; other language cases protect compatibility and tokenization regressions.

Capture-intelligence changes should run:

npm run benchmark:capture

The default profile preserves the v0.5.0 exact-duplicate baseline for reproducibility. The current bounded relationship contract has a separate explicit gate:

npm run benchmark:capture -- --expect bounded

The capture benchmark uses public synthetic data and independently reports English relationship quality, policy blocks, scope and archived isolation, candidate/evidence bounds, latency, and zero memory or audit writes. The bounded profile shipped in 0.6.0 and remains the regression gate for that contract.

Relation read and governance changes should also run:

npm run benchmark:relations
npm run benchmark:governance

The first gate proves ordered response equivalence while bounding relation query growth at recall/list limits 50 and 200. The second repeats a content-free read-only governance report over sparse and relation-dense stores, checking stable ordering, bounded latency, expected candidate counts, and unchanged memory, relation, and audit-event counts.

Optional-semantics work should also run:

npm run benchmark:semantics
npm run benchmark:semantics -- --store-size medium

This benchmark compares FTS, a deterministic offline semantic candidate, and hybrid reciprocal-rank fusion. It reports the English quality bar separately from compatibility cases and treats scope isolation, archived exclusion, bounded output, zero writes, and no network as independent safety gates. The medium-store profile also gates scoped status, fallback, cold and warm hybrid recall, peak RSS, and authorized vector row counts. The benchmark encoder is test evidence, not a bundled runtime provider.

Smoke tests may set NUZO_DOCTOR_SKIP_GIT=1 so restricted environments do not warn only because Git tracking inspection is unavailable. Tests should still cover normal warning behavior for missing stores and tracked memory files.

MCP doctor tests should cover both ok and warning states. They must assert that diagnostics include aggregate counts, store readability/writability, and schema status, but not memory content.

Continuous Integration

GitHub Actions runs validation from:

.github/workflows/ci.yml

The workflow runs on pull requests, pushes to main, and manual dispatch.

Node validation uses the lockfile on Node.js 22 LTS and 24 LTS and runs:

npm ci
npm run check
npm run release:check
npm test
npm run build
npm run benchmark:recall
npm run benchmark:capture
npm run benchmark:capture -- --expect bounded
npm run benchmark:relations
npm run benchmark:governance
npm run benchmark:semantics
npm run benchmark:semantics -- --store-size medium
npm run package:plugins
npm run validate:npm
npm run smoke:cli
npm run smoke:host-hooks
npm run smoke:claude-code-plugin
npm run smoke:codex-plugin

The one-line installer has an optional Docker smoke gate:

npm run smoke:installer

It runs docs/install.sh in ephemeral Linux containers with representative POSIX shells and Node/npm states. Deterministic success scenarios use fake npm and nuzo commands so the gate proves the installer invokes npm install --global @nuzo/memory@... without configuring Codex or Claude Code. Release scenarios also perform a real global npm install inside Node 22 and Node 24 containers for the current public package. Failure scenarios cover missing Node.js, unsupported Node.js, unsupported npm, and invalid installer versions. Containers use --rm and a Nuzo-specific label; the harness removes only containers with that label if cleanup is needed.

On Node.js 24, CI additionally installs the tracked repository marketplaces with pinned Codex and Claude Code CLIs and runs npm run smoke:host-canary. Published-package smokes remain post-release checks because they require the new version to exist in the npm registry.

Published host canaries create their SQLite fixture with the published CLI and inspect post-hook history with that same runtime. This keeps the source schema out of a public-version continuity test and prevents false failures after a new migration lands on main.

Installed artifact validation also runs a cost-conscious OS/architecture matrix on Node.js 22 LTS and 24 LTS:

ubuntu-latest
macos-15-intel
windows-latest

Each matrix lane runs:

npm ci
npm run smoke:os-artifacts

smoke:os-artifacts builds staged npm tarballs, installs the staged active @nuzo/memory-core, @nuzo/memory, and, from 1.1.0, @nuzo/memory-mcp packages into a temporary project, exercises CLI session continuity, MCP stdio session continuity, installed host hook doctor diagnostics, and generated Codex and Claude Code plugin command forms. This is the release-blocking OS matrix documented in Runtime Support.

Manual dispatch can also run a release rehearsal job. It uses Node.js 24 and executes:

npm run release:rehearse -- X.Y.Z

Manual dispatch requires an explicit rehearsal version.

Release-state validation confirms that root/workspace versions, Nuzo internal dependency pins, lockfile workspace entries, CLI version output, MCP server version metadata, and changelog structure stay aligned.

The npm artifact validation installs generated core and Nuzo package tarballs into a temporary project. It exercises the installed CLI workflow, read-only capture suggestions, confirmed writes, recall, and exit codes, then confirms the installed MCP binary starts and supports the same suggestion-to-recall lifecycle over stdio. It does not publish packages or require npm credentials.

Documentation Drift Gates

User-facing documentation is part of the release contract. CI and Pages run:

npm run docs:check

That command combines three bounded gates:

1. tools/docs-contracts.test.mjs derives release versions, MCP tool names and count, supported Node.js lines, current-versus-upcoming setup language, npm package guidance, and primary navigation boundaries from canonical sources.
2. tools/docs-link-check.mjs validates Markdown local links, anchors, and deduplicated external links with a short timeout and an explicit ignore policy for dynamic historical evidence such as old GitHub Actions run URLs and generated badge images.
3. tools/docs-snippet-check.mjs validates shell snippets with bash -n without executing them, keeps first-use setup/update snippets aligned across README, docs, and npm README entry points, and protects a small set of public product terms such as Nuzo, Codex, and Claude Code.

Use NUZO_DOCS_LINK_CHECK_EXTERNAL=0 npm run docs:links only for local diagnosis when the network is unavailable. Do not use that override in CI or as release evidence. If an external site is rate-limited, historical, or dynamic, add the narrowest ignore rule to tools/docs-link-check.mjs with a comment or surrounding code that explains the category. Do not ignore ordinary broken documentation links.

The recall benchmark is a required CI gate, not only a local release check. It protects the canonical FTS relevance envelope before host lifecycle tests reuse recall results. The capture benchmark runs twice in CI: the default baseline profile preserves the historical exact-duplicate behavior, while --expect bounded is the current bounded relationship contract gate. The relation hydration and governance benchmarks independently gate query growth, output equivalence, deterministic content-free candidate ordering, bounded dense-store latency, and proof that the governance report writes no store state.

The host hook continuity smoke uses an isolated SQLite store populated only with synthetic data. Its 0.2.1 matrix covers 75 memories and 53 scenarios, including global and project scopes, autoload bootstrap, exact topical tags, archived and unrelated records, bounded output, fail-open input handling, common preference/fact/instruction/decision workflows, and prompts in 14 languages (English plus 14 non-English languages). Representative ephemeral Codex sessions complement this deterministic test before release; Claude Code remains covered by artifact and schema validation when its authenticated host CLI is unavailable locally.

The NUZO-37 host canary is the smallest cross-host lifecycle proof. It creates one fake user:default instruction memory tagged autoload, runs separate fresh SessionStart hook invocations through the generated Codex and Claude Code plugin artifacts against the same SQLite store, and asserts both hosts receive the same attributed record inside the untrusted-memory boundary. It also checks that unrelated UserPromptSubmit recall does not repeat the autoload canary and that hook execution creates no audit writes. This proves Nuzo delivery, not model obedience: hosts may use or ignore recalled memory according to their own instruction hierarchy.

For host-native packaging confidence, run:

NUZO_HOST_CANARY_NATIVE=1 npm run smoke:host-canary

This additionally installs the tracked Nuzo marketplace and plugin under an isolated CODEX_HOME and isolated Claude Code home. Both hosts must discover and enable nuzo@nuzo-memory; Claude Code also validates the repository marketplace schema. It still separates plugin and hook delivery from LLM response compliance.

The published CLI and MCP smokes install @nuzo/memory into a temporary npm prefix and validate session continuity through separate nuzo and stdio server processes. The published optional-semantics smoke validates that a default install contains no Transformers.js runtime or model files, that hybrid recall falls back visibly without a sidecar, and, when NUZO_SEMANTIC_MODEL_PATH is set, that the published CLI can install the exact optional peer, rebuild a derived sidecar, and recall a paraphrased memory through semantic and hybrid mode. These are post-release confidence checks for package users and host plugin installs, not replacements for staged artifact validation before publication.

Starting with 1.1.0, the staged npm matrix also installs @nuzo/memory-mcp, verifies its canonical mcpName and single executable, and opens an MCP SDK session through memory-mcp. npm run registry:check guards the repository, package, and version boundary. npm run registry:validate adds exhaustive validation with the checksum-pinned official Registry publisher; it never authenticates or publishes.

The Codex plugin artifact smoke regenerates the release-layout plugin, verifies the host-facing metadata loads as Nuzo, reads its bundled MCP config, and validates continuity through staged npm tarballs before publication. It still asserts the version-pinned command in the generated config. After publication, NUZO_PLUGIN_SMOKE_PUBLISHED=1 npm run smoke:codex-plugin validates the exact public command that the plugin exposes. Published hook canaries launch from the repository root, while npm resolution is anchored at the plugin root, so a user's npm workspace cannot shadow the pinned runtime. Codex MCP uses . as a plugin-relative cwd; hook commands use ${CLAUDE_PLUGIN_ROOT}.

The continuity assertion covers the 0.2.0 Codex task-start recall contract: a first MCP session writes a fake confirmed memory, a later session calls memory.recall_hook, the remembered content is returned, and the hook reports memory_writes: false plus capture_suggestions: false without adding recall audit events.

The same smoke covers the inferred capture confirmation contract: a ready memory.suggest_capture draft is not visible to recall before confirmation, a rejected draft is never persisted, a confirmed draft is written through memory.confirm_capture, and a later equivalent suggestion is reported as a duplicate.

For update safety, the smoke also updates a confirmed capture with memory.confirm_capture, target_memory_id, and expected_revision, then verifies a stale update returns MEMORY_REVISION_CONFLICT instead of overwriting the newer memory.

The Claude Code plugin artifact smoke performs the same release-layout continuity validation while also resolving the ${CLAUDE_PLUGIN_ROOT} cwd placeholder against an isolated temporary plugin install path. Set NUZO_PLUGIN_SMOKE_PUBLISHED=1 after release to exercise its exact public version-pinned command.

Documentation validation installs requirements-docs.txt and runs:

mkdocs build --strict

CI does not deploy GitHub Pages and only receives read access to repository contents.

The matrix is the supported Node.js policy for the MVP. The package.json engine declarations provide a minimum version guard, while CI defines the major versions Nuzo actively claims and tests.

npm audit --audit-level=moderate remains a dependency-change and release gate. Advisory changes outside the repository should not make ordinary pull requests nondeterministically fail.