bethel-nz
/Greppa
/project2026
/project2026

2026

Greppa

Chat that keeps going after you close the tab, with memory kept separate for each user.

Visit project ↗
Role
Per-tenant AI memory + resumable streaming
Problem
Close the tab mid-answer and the work dies with it. Shared vector indexes make privacy a filter someone has to remember forever.
Constraint
Keep generating after disconnect; resume without gaps or double tokens; never open another scope's memory file.
Outcome
Ordered event log for resume. One SQLite file per scope, hydrated and published through Checkpoint (copy-on-write + ETag).

The two problems it solves

  1. Once a message is accepted, closing the browser should not cancel the work.
  2. Personal memory should belong to someone because of how it is stored — not because every query remembered a tenant filter.

Chat became a durable stream: worker keeps going, client reconnects with the last sequence it saw, server replays the rest. No second answer, no silent hole in the middle.

Memory went the other way from “one big index.” Each user or org scope gets its own SQLite file. Checkpoint pulls it from R2 when needed, readers pin an immutable generation, writers publish under ETag. Another person's data is not filtered out after search — their file never opens.

This is still a foundation. Streaming and scoped files first; not a finished multi-tenant AI product.

What breaks in the usual shape

Long-lived HTTP chat ties acceptance, model run, token delivery, and recovery to one connection. Refresh, laptop sleep, LB timeout — the only live path to the answer is gone.

Shared vector + filter is fast until one path forgets the filter. That is a convention, not a wall.

So Greppa splits the job: API accepts → workflow runs → Redis event log records → Checkpoint opens the right memory file. UI can be flaky. Those pieces should not be.

A stream that survives the client

Chat is not one long HTTP request. POST /chat validates, stores the user message, returns 202 + messageId. A durable workflow owns the rest.

Closing the tab can stop delivery. It should not cancel accepted work. Queue retries are normal; a redelivered run checks terminal state first so you do not get two answers for one message.

Event log, not “just tokens”

The UI needs search cues, sources, completion, and errors — not only token text. Those are typed events (cue, sources, token, done, error) with a monotonic sequence.

Each event is written to a Redis sorted set before it is broadcast to the live channel:

typescript
1export function makeEmitter({ messageId, ttlMs }: { messageId: string; ttlMs: number }) {
2 const channel = realtime.channel(messageId)
3 const eventsKey = `msg:${messageId}:events`
4 const ttlSecs = Math.floor(ttlMs / 1000)
5 let seq = 0
6
7 return async function emit(type: EmitType, data: unknown): Promise<StoredEvent> {
8 seq += 1
9 const event: StoredEvent = { id: String(seq), seq, type, data }
10
11 await redis
12 .pipeline()
13 .zadd(eventsKey, { score: seq, member: JSON.stringify(event) })
14 .expire(eventsKey, ttlSecs)
15 .exec()
16
17 await channel.emit(`msg.${type}` as EmitEvent, event as any)
18 return event
19 }
20}

Order matters more than brand names. Realtime is the fast path; Redis is the log you can replay. Broadcast dies, events stay. done is written before the message is marked complete — “finished” without that event is wishful thinking.

Reconnect race

The SDK only advances the cursor after a full SSE frame. Reconnect sends last-event-id; server replays after that. Bad cursor → full replay. Duplicates you can drop; missing events you cannot invent.

The annoying part was joining history to live. Read Redis first → miss events in the gap. Forward live immediately → new tokens ahead of old replay. The interactive below is how I checked the only order that stayed safe:

The dangerous moment is the handoff.

Change the join order and watch which events survive the seam between replay and live delivery.

Redis ZSET

snapshot

123456

Realtime

live tail

—buffer []

SSE client receives

id = seq
············

ready

Fig.Run each join order. Only buffering before the snapshot and draining after replay delivers 1–12 without a gap.

The final route sends all three sources—stored history, realtime events caught during replay, and the live tail—through one sequenced writer. That gate turns overlap into harmless duplicates instead of duplicate SSE frames. The handoff in routes/chat/stream.ts is:

typescript
1let lastSeq = 0
2let terminated = false
3
4const forward = async (ev: StoredEvent) => {
5 if (terminated || ev.seq <= lastSeq) return
6
7 lastSeq = ev.seq
8 await stream.writeSSE({
9 id: String(ev.seq),
10 event: ev.type,
11 data: JSON.stringify(ev.data),
12 })
13
14 if (TERMINAL.has(ev.type)) terminated = true
15}
16
17// Listen before reading Redis so an event cannot land in the gap.
18const buffer: StoredEvent[] = []
19let snapshotDone = false
20
21const unsubscribe = await realtime.channel(messageId).subscribe({
22 events: ['msg.cue', 'msg.sources', 'msg.token', 'msg.done', 'msg.error'],
23 history: false,
24 onData: (envelope: any) => {
25 const ev = envelope.data as StoredEvent
26 if (!snapshotDone) buffer.push(ev)
27 else void forward(ev)
28 },
29})
30
31try {
32 // 1. Replay the durable snapshot after the client's cursor.
33 const rawLog = (await redis.zrange(eventsKey, 0, -1)) as unknown[]
34 const log: StoredEvent[] = rawLog.map((item) =>
35 typeof item === 'string' ? (JSON.parse(item) as StoredEvent) : (item as StoredEvent),
36 )
37 const maxSeq = log.length ? log[log.length - 1].seq : 0
38 const effectiveCursor = Number.isFinite(cursor) && cursor <= maxSeq ? cursor : 0
39
40 for (const ev of log) {
41 if (ev.seq > effectiveCursor) await forward(ev)
42 if (terminated) break
43 }
44
45 // 2. Forward arrivals captured while Redis was being replayed.
46 while (buffer.length && !terminated) await forward(buffer.shift()!)
47
48 // 3. The queue is empty; new arrivals can flow straight to forward().
49 snapshotDone = true
50 if (terminated) return
51
52 // Keep the SSE request open while realtime calls forward(). If the worker
53 // disappears without done/error, close the stream at the resume boundary.
54 let seen = lastSeq
55 let lastActivity = Date.now()
56 while (!terminated) {
57 await new Promise((resolve) => setTimeout(resolve, 50))
58 if (lastSeq !== seen) {
59 seen = lastSeq
60 lastActivity = Date.now()
61 } else if (Date.now() - lastActivity > cfg.resumeWindowMs) {
62 await stream.writeSSE({
63 event: 'error',
64 data: JSON.stringify({ code: 'stalled', reason: 'stream stalled' }),
65 })
66 terminated = true
67 }
68 }
69} finally {
70 unsubscribe()
71}

Four stages: capture → replay → drain → live tail, one high-water sequence through all of them. UI just sees a continuous answer. incomplete / stalled stop infinite waiting when a worker dies or the log expires.

One resume window covers the event log, metadata, and stall timeout. Same policy, not three timers that drift.

The tab can leave. The turn keeps going.

Watch the client disconnect while the workflow continues to append to the event log.

greppa.chat

Browser client

cursor —
Why does closing this tab not stop the answer?

Durable worker

idle

QStash

workflow

Event log

Realtime

no subscriber

produced 0 · delivered 0

replay 0

Fig.Run a Greppa turn: POST chat returns a message ID, QStash continues the workflow, Redis persists sequenced events, and a reconnect replays after the browser cursor.

Memory with an owner

Delivery alone is not enough. I wanted isolation in the storage shape: one portable file per scope — text, embeddings, indexes. Postgres tracks ownership; the memory path only ever opens the file for that scope.

First engine lied quietly

First version used Memvid. No crash. has_vec_index: true with zero vectors. “Semantic” search was keyword overlap. "feline pet animal" vs a doc about a pet mammal → nothing, for as long as the feature had existed.

Numbers made the swap easy: 350 short notes (~700 KB text, no vectors) → 16.5 MB file; ~600 notes hit a 50 MB ceiling. Closed binary, hang with no timeout on one write path. Decorating that felt wrong.

Replacement: SQLite + sqlite-vec + FTS5, fused with reciprocal rank. Same corpus: 80s → 0.18s ingest, 16.5 MB empty-index → 7.6 MB with embeddings, single-digit ms answers. The metric that mattered: queries with no shared words still find the doc.

Checkpoint between R2 and SQLite

One file per scope still needs a lifecycle. Download every request is slow; keep every file open forever blows disk. Checkpoint is a bounded, refcounted working set: what stays warm, which generation a reader sees, when a write is safe to publish.

Race case — search while ingest writes the same scope:

The generation is immutable. The lock is brief.

Race a long read against a write and follow one scope from object storage into the open-file map.

scope key
mutex · free

Cloudflare R2

scopes/usr_ada/memory.sqlite

object version

v1

hydrate

Checkpoint.open Map

/tmp/greppa/usr_ada/memory.sqlite

etag

v1

refcount

0

exists

true

read(key, fn)

idle

write(key, fn)

waiting
Fig.The read keeps generation v1 after releasing the lock; the write seals and conditionally commits v2.

Only Checkpoint deals with the file mess

App code does not touch locks, temp files, or ETags. It runs a callback on a local path. Store logic stays in the callback; lifecycle stays in Checkpoint. That is why swapping Memvid → SQLite did not rewrite Checkpoint:

typescript
1export type CheckpointConfig = {
2 storage: StorageBackend
3 cacheDir: string
4 maxOpen: number
5 maxCacheBytes?: number
6 idleMs: number
7 now?: () => number
8}
9
10export declare class Checkpoint {
11 constructor(config: CheckpointConfig)
12
13 get openCount(): number
14
15 read<T>(
16 key: string,
17 useSnapshot: (localPath: string) => Promise<T>,
18 ): Promise<T>
19
20 write<T>(
21 key: string,
22 update: (localPath: string, exists: boolean) => Promise<T>,
23 ): Promise<T>
24
25 stat(key: string): Promise<ObjectMeta | null>
26 list(prefix: string): Promise<ObjectMeta[]>
27 delete(key: string): Promise<void>
28
29 startEviction(intervalMs?: number): void
30 stopEviction(): void
31 evictIdle(): Promise<void>
32 closeAll(): Promise<void>
33}

On a read, Checkpoint hydrates a cold scope file and hands the callback an immutable generation. It no longer copies the file per read—an earlier version did, which meant every question paid to duplicate the entire memory. Because writers always build a separate generation, readers can share one without a copy and without a lock.

On a write, the callback mutates a private working copy and closes it. Checkpoint renames that into a candidate and uploads it under an ETag condition, so an interrupted mutation cannot publish a partial file. If the condition fails, the losing writer does not overwrite the winner and does not discard its own work: the stale candidate is deleted, the winning version is hydrated, and the mutation is re-run against it. Two writers appending to one scope both end up in the result rather than one silently replacing the other.

The working set stays bounded by maxOpen, a byte budget, and an idle timeout. Reference counts keep active files resident, and eviction will exceed its budget rather than delete a file a reader is still using—an overage the process reports instead of resolving by force.

The honest limits are elsewhere. Every write re-uploads the entire scope file, so appending a 2 KB note to a 16 MB scope moves 16 MB; the local insert takes half a millisecond and the upload takes seconds. Vector search is exact rather than approximate, which is a recall advantage but scales linearly, so a single very large scope would need an index this design does not have. And a cold read pays for the whole file before it can answer, because Checkpoint is a whole-object cache and not a virtual filesystem.

The generation is immutable. The lock is brief.

Race a long read against a write and follow one scope from object storage into the open-file map.

scope key
mutex · free

Cloudflare R2

scopes/usr_ada/memory.sqlite

object version

v1

hydrate

Checkpoint.open Map

/tmp/greppa/usr_ada/memory.sqlite

etag

v1

refcount

0

exists

true

read(key, fn)

idle

write(key, fn)

waiting
Fig.Race Checkpoint read and write on one scope key: the read pins immutable generation one and releases the mutex before the write seals and conditionally commits generation two.

What holds, what does not

Working today:

  • accept, then keep generating if the browser leaves
  • resume from last sequence without inventing a second answer
  • cues / sources / tokens / done / error as real product states
  • open only the scope file that belongs to the request
  • readers pin a generation; writers CAS-publish; losers re-run on the winner
  • local working set bounded by open count and bytes

Still rough:

  • every write re-uploads the whole scope file — fine for personal scale, bad for big docs
  • resume needs more real infra abuse, not just mocks