How to Build Agent Memory That Finds What You Mean (Even When You Misspell It)

Spectral keeps a cross-session memory of your project: small, durable observations written down when a session ends, queryable with read_project_observations before the next session touches any code. For a long time that memory had an embarrassing secret — you could only find things in it by exact substring. The tool's own description used to apologize for it:

"Uses simple substring matching — be specific with your query terms."

We rebuilt the search underneath it: n-gram indexing, ranked scoring, typo tolerance, and near-duplicate detection on the write side. No new dependencies, no embedding service, no vector database.

The failure you've probably hit#

A session in March writes:

Prisma migrations live in backend/prisma/migrations — rollback with prisma migrate resolve, never by hand.

Two weeks later the agent asks its memory: "prisma migrate rollback". The old search ran:

WHERE content LIKE '%prisma migrate rollback%'

Every word is there; the exact phrase is not. 0 results — and the agent hand-edits a migration, the exact mistake the memory existed to prevent. One letter off is just as dead: "migraton" returned nothing too.

Trigrams: substring search you already ship#

Agent queries are natural-language fragments, not phrases — so the right semantics is an AND of query tokens, ranked, with typo tolerance. The trick is an old one: an n-gram index with n = 3. FTS5 ships a tokenize='trigram' option inside the SQLite builds Spectral already uses, so the whole feature is one virtual table:

CREATE VIRTUAL TABLE project_observations_fts USING fts5(
  content, project_id UNINDEXED, obs_id UNINDEXED,
  tokenize='trigram'
);

Every 3-character window gets indexed, so any substring becomes findable — including mid-word matches that broke plain full-text search for code identifiers. The index is synced in the same transaction as every write and backfilled on open, so it can never drift from the rows. A query becomes MATCH '"prisma" AND "migrate" AND "rollback"': case-insensitive substring matches in any order, any position. Sub-3-character tokens fall through to a plain LIKE filter with the same semantics.

Rank, don't just list#

Finding 20 rows is easy; ordering them is what makes the agent trust the tool. The old search sorted by date. The new one scores:

score = 0.6 × text match (bm25)
      + 0.25 × relevance tag
      + 0.15 × recency

bm25 is mapped into 0–1, relevance is the write-time tag (critical 1.0 → low 0.25), and recency is e^(−ageDays/30) — this week's memory softly outranks last quarter's without hiding it.

Typos and clean writes#

The AND-match has a weakness: one misspelled token zeroes the whole query. So "0 hits" is treated as a signal, not an answer — the search re-scans and scores with character-trigram Jaccard, token by token. "migraton" vs "migrations" shares 4 of 9 windows (≈ 0.44); the gate is ≥ 0.2 per token and mean ≥ 0.3. So "migraton" is found, while "eslint" still doesn't match Prisma notes. Tolerance with precision.

The write side gets the same treatment. Years of near-identical rewordings bury signal under echo, so every write runs a cheap trigram-similarity check: ≥ 0.9 Jaccard → the old wording is replaced; below that, appended. Identical content still maps to the same id, so re-writes stay idempotent. One honest trade-off: a rewording gets a new id, so previously shared ids may become unresolvable — a memory that deduplicates is worth more than one where stale ids live forever.

Takeaways if you're building agent memory#

1. Substring search is the wrong default for natural-language queries; an AND-of-terms match is the floor.
2. FTS5 trigram gives substring semantics plus bm25 with zero infrastructure — it's in the SQLite you already ship.
3. Always rank. Recency-only ordering trains the agent to ignore the tool.
4. Gate your fallbacks. Thresholds (0.2 per token, 0.3 mean, 0.9 dedup) make tolerance precise instead of fuzzy.
5. Deduplicate at write time. Not polluting memory is cheaper than cleaning it.

A few hundred lines, one virtual table, one scoring function — and it's the difference between a memory the agent distrusts and one it checks before every edit.