Streaming outbox events
NonoDB can tail append-only event collections in two ways:
watch— a bounded long-poll request over/v1/exec;/v1/watch— a continuous NDJSON or SSE stream.
Both are designed for collections such as _events, where documents are
inserted with sortable ULID _id values by the outbox producer.
The normal mode tails existing documents by _id. It is not an automatic
change stream for arbitrary collections; updates and deletes are not emitted.
For change notification, write an event document explicitly with
EventProducer, or use the connection-scoped
live mode.
Live subscriptions (connection-scoped)
Section titled “Live subscriptions (connection-scoped)”POST /v1/watch also supports mode:"live". This does not tail _events.
It registers an in-memory subscription for the lifetime of the HTTP connection,
evaluates a condition in the writer, and pushes matching changes directly to the
stream.
{ "op": "watch", "coll": "orders", "mode": "live", "filter": {"status": "paid"}, "ops": ["insert", "update", "delete"], "heartbeat_ms": 10000}Optional live fields:
| Field | Meaning |
|---|---|
filter |
Condition on the after document (insert/update) or before document (delete) |
ops |
insert/update/delete; empty means all |
count |
Count condition over current collection state, e.g. {"filter":{"total":{"$gt":100}},"gt":100} |
once |
Optional edge-triggered mode (false -> true); default emits every matching write |
name |
Subscription name used in the default event type |
event_type |
Optional delivered event type; default subscription.<name> |
heartbeat_ms |
Server heartbeat, default 10,000, max 10,000 in live mode |
timeout_ms |
Optional maximum stream lifetime |
Live events look like ordinary subscription events:
{ "_id": "01J...", "type": "subscription.live", "doc_id": "o1", "payload": { "subscription": "live", "coll": "orders", "op": "update", "before": {"_id": "o1", "status": "new"}, "after": {"_id": "o1", "status": "paid"}, "count": null }}Semantics:
- the subscription exists only while the stream is connected;
- closing the stream removes it immediately;
- a janitor removes half-open connections after
30seconds without a successful heartbeat; - live events are not written to
_events, so they do not grow the database; - there is no replay: after a disconnect, run a snapshot/
findand then open a new live stream if needed; - slow consumers are disconnected instead of buffering without bound;
after,types, andthresholdare rejected in live mode;- vault collections are rejected;
- SQL writes,
find_and_modify, TTL expiry, and normal writes all trigger live evaluation.
Python:
for event in db.collection("orders").watch_live( filter={"status": "paid"}, heartbeat_ms=10_000,): print(event["payload"]["after"])Because the subscription is tied to the connection, watch_live does not have a
resume cursor. Reconnect creates a fresh subscription.
Long-poll request
Section titled “Long-poll request”{ "op": "watch", "coll": "_events", "after": "01J...", "limit": 100, "timeout_ms": 1000, "types": ["order.paid"], "filter": {"payload.total": {"$gte": 1000}}, "projection": {"type": 1, "payload": 1}}| Field | Meaning |
|---|---|
coll |
Append-only event collection |
after |
Last processed event _id; omit to start from the beginning |
limit |
Maximum events in this batch, default 100, max 1,000 |
timeout_ms |
Long-poll timeout, default 1,000, max 25,000 |
types |
Optional event type allowlist |
filter |
Optional MongoDB-style filter on event documents |
projection |
Optional projection; _id is always retained for resume |
The request is a normal read operation. It requires read access to coll, is
not cached, and is not allowed inside a read batch. Vault collections are
rejected.
Long-poll response
Section titled “Long-poll response”{ "events": [ { "_id": "01J...", "type": "order.paid", "doc_id": "o1", "payload": {"total": 1200}, "created_at": 1762800000000 } ], "next": "01J...", "timed_out": false, "has_more": false, "truncated": false, "oldest": null}| Field | Meaning |
|---|---|
events |
Events ordered by _id |
next |
Resume token for the next request |
timed_out |
True when no event arrived before timeout_ms |
has_more |
True when another page is immediately available |
truncated |
True when after is older than retained events |
oldest |
Oldest retained event id when truncated is true |
When has_more is true, call again with after: next. When it is false, keep the
same cursor and call again; the request will long-poll for new events.
truncated means the server could not confirm that after still exists. Events
between after and oldest may have expired or been deleted. A consumer that
needs no gaps should restart from oldest after a snapshot/rebuild, not silently
continue.
Continuous stream
Section titled “Continuous stream”POST /v1/watch uses the same request body and returns either:
application/x-ndjsonby default — one JSON object per line;text/event-streamwhen the request sendsAccept: text/event-stream.
curl --unix-socket /run/nonodb/nonodb.sock \ -H 'content-type: application/json' \ -H 'accept: application/x-ndjson' \ -d '{"op":"watch","coll":"_events","after":"01J..."}' \ http://localhost/v1/watchNDJSON frames:
{"stream":{"truncated":false,"oldest":null}}{"_id":"01J...","type":"order.paid","payload":{"total":1200}}{"heartbeat":true,"ts_ms":1762800001000}SSE frames:
event: readydata: {"truncated":false,"oldest":null}
event: messageid: 01J...data: {"_id":"01J...","type":"order.paid","payload":{"total":1200}}
event: heartbeatdata: {"heartbeat":true,"ts_ms":1762800001000}Stream fields:
| Field | Meaning |
|---|---|
heartbeat_ms |
Keepalive interval, default 15,000, range 100..60,000 |
timeout_ms |
Optional maximum stream lifetime; 0/absent means no limit |
overlap_ms |
Late-commit rescan window, default 2,000, max 60,000; 0 disables |
filter |
Optional MongoDB-style filter |
projection |
Optional projection; _id is always retained |
limit |
Batch size per internal poll, default 100, max 1,000 |
Heartbeats keep idle connections alive; they are not events. The stream stays open until the client disconnects, the maximum lifetime expires, or the daemon shuts down.
Concurrent producers may commit a lower _id after a higher one has already
been delivered. The stream rescans the last overlap_ms window, suppresses
already-delivered ids in memory, and can therefore deliver a lower id after a
higher id. Event order is not a global commit order; delivery remains
at-least-once. The bounded long-poll watch does not do this rescan — use the
continuous stream or the Python EventConsumer when producers run concurrently.
The response is not cached and not audited as an ordinary request-sized
operation; audit logs record watch_stream when the stream closes.
Python
Section titled “Python”from nonodb import NonoDB, StreamTruncatedError
db = NonoDB()events = db.collection("_events")
# One batch, useful for manual checkpointing.batch = events.watch_once( after=None, limit=100, timeout_ms=1000, types=["order.paid"], filter={"payload.total": {"$gte": 1000}}, projection={"type": 1, "payload": 1},)for event in batch["events"]: process(event) # must be idempotentcursor = batch["next"]
# Infinite long-polling generator with reconnects.for event in events.watch(after=cursor, timeout_ms=1000, retry=True): process(event) if should_stop: break
# Continuous NDJSON stream with reconnects.try: for event in events.watch_stream( after=cursor, heartbeat_ms=15000, overlap_ms=2000, filter={"type": "order.paid"}, retry=True, ): process(event)except StreamTruncatedError as exc: # Rebuild from exc.oldest or take a fresh snapshot. raisewatch_once returns the full response object. watch and watch_stream yield
event dictionaries, advance their local cursor, and reconnect after transport
errors. Persist your own checkpoint after successful processing. Both generators
raise StreamTruncatedError when the server reports a gap.
use nonodb_client::{Client, Error};use serde_json::json;
let db = Client::from_env()?;let events = db.collection("_events");let batch = events.watch_with( None, 100, 1000, &["order.paid"], Some(&json!({"payload.total": {"$gte": 1000}})), Some(&json!({"type": 1, "payload": 1})),)?;for event in &batch.events { process(event)?;}if batch.truncated { eprintln!("gap before {:?}", batch.oldest);}let next = batch.next;The Rust client exposes the long-poll API. For a continuous stream, POST the same
body to /v1/watch and read NDJSON lines from the response.
Node.js
Section titled “Node.js”const { NonoDB, NonoDBError } = require('./client/node');
const db = new NonoDB();const events = db.collection('_events');
const batch = await events.watchOnce(null, { limit: 100, timeoutMs: 1000, types: ['order.paid'], filter: { 'payload.total': { $gte: 1000 } }, projection: { type: 1, payload: 1 },});for (const event of batch.events) await process(event);const next = batch.next;
// Async generator with reconnect and truncation detection.try { for await (const event of events.watch(next, { timeoutMs: 1000 })) { await process(event); }} catch (error) { if (error instanceof NonoDBError && error.code === 'stream_truncated') { console.error('gap before', error.oldest); } throw error;}package main
import ( "context" "fmt"
nonodb "github.com/essmos/nonodb/client/go")
func main() { ctx := context.Background() db, _ := nonodb.New() events := db.Collection("_events")
batch, err := events.Watch(ctx, nonodb.WatchOptions{ Limit: 100, TimeoutMs: 1000, Types: []string{"order.paid"}, Filter: map[string]any{"payload.total": map[string]any{"$gte": 1000}}, Projection: map[string]any{"type": 1, "payload": 1}, }) if err != nil { panic(err) } if batch.Truncated { fmt.Printf("gap before %s\n", batch.Oldest) } for _, event := range batch.Events { fmt.Println(event["_id"], event["type"]) } _ = batch.Next}The Go client exposes the long-poll API. For a continuous stream, POST the same
body to /v1/watch and decode NDJSON lines with json.Decoder.
Checkpoints, delivery, and retention
Section titled “Checkpoints, delivery, and retention”watch and /v1/watch are transport-level streams, not durable consumers. They
do not store a checkpoint. The application must persist the last processed
_id after successful processing, just like the
outbox consumer.
Delivery is at-least-once:
- a crash after processing but before saving the cursor replays the batch;
- handlers must be idempotent;
- a client restart should resume from its stored cursor;
- replaying retained events is safe for idempotent handlers.
Apply TTL to the event collection so history does not grow forever:
db.create_collection("_events", ttl_seconds=7 * 24 * 3600)# If it already exists:db.set_collection_ttl("_events", 7 * 24 * 3600)Choose retention longer than the maximum consumer outage. An expired event
cannot be replayed; watch reports that condition as truncated.
For the higher-level Python consumer with an overlap rescan for late-committed
lower ids, use EventConsumer from nonodb_worker.
Event thresholds
Section titled “Event thresholds”The continuous /v1/watch stream can collapse many matching events into one
threshold event. This keeps ordinary events in the outbox but reduces the number
of frames sent to the consumer.
{ "op": "watch", "coll": "_events", "types": ["orders.insert"], "threshold": { "count": 100, "window_ms": 60000, "group_by": ["payload.tenant"] }}Semantics:
countmust be positive;thresholdis ignored when it is0or absent.- Events are counted per
group_bykey and perwindow_mswindow. - When a group reaches
count, the triggering event is sent with athresholdfield and all later matching events in that window are suppressed. - The window resets after
window_ms, so a later window can emit again. group_bypaths are read from the event document; missing values group asnull.- Threshold state is in memory for one stream connection. Reconnecting starts a new window. Delivery remains at-least-once and is not a replacement for durable business counters.
- Long-poll
watchdoes not supportthreshold; use/v1/watch.
for event in events.watch_stream( types=["orders.insert"], threshold={"count": 100, "window_ms": 60_000, "group_by": ["payload.tenant"]},): print(event["threshold"], event["payload"])Metrics
Section titled “Metrics”/metrics exposes stream counters:
| Metric | Meaning |
|---|---|
watch_streams_total |
Continuous /v1/watch connections opened |
watch_active |
Currently open continuous streams |
watch_events_total |
Events delivered by long-poll and streams |
watch_timeouts_total |
Long-poll requests that returned no event |
watch_truncated_total |
Watch requests with an expired/unknown resume cursor |
watch_heartbeats_total |
Keepalive frames sent |
watch_threshold_emitted_total |
Threshold events sent |
watch_threshold_suppressed_total |
Events suppressed by thresholds |
collection_events_emitted_total |
Automatic collection events written |
collection_events_suppressed_total |
Events suppressed by source-side filters |
Validation
Section titled “Validation”python3 workers/python/tests/events_integration.pyThe integration test covers the first batch, empty long-poll timeout, an event
arriving during a long poll, type filtering, payload filtering, projection,
pagination with has_more, truncation detection, the NDJSON and SSE streams,
the Python generators, and concurrent stream delivery (6 producers × 100 events,
6 independent consumers, no missing or duplicate events).
For a larger repeatable run:
PRODUCERS=16 PER=1000 CONSUMERS=16 \ python3 workers/python/tests/stream_stress.pyThe latest stress run delivered 16,000 events to 16 independent stream
consumers, with 0 missing and 0 duplicate deliveries. watch_active returned to
0 after the consumers disconnected.