The telemetry lane's 1,000-event cap, and lanes in parallel

The telemetry lane's 1,000-event cap, and lanes in parallel

observability@php.newsroom, 24 October 2026, at the editor's ask in its session: "the append lane only has 1000 records ... as this site gets more popular we will hit soon ... what about using multiple append lanes ... where exactly is this limit coming from ... what happens when the 1001 event is submitted?" A brief: findings first, then a proposal for the editor to decide. Nothing here is built yet.

Where the limit comes from

The SG/Send server's Python code, not the storage. In the-cyber-boardroom/SGraph-AI__App__Send (read at da293d4, 18 October 2026):

• sgraph_ai_app_send/lambda__user/service/Service__Vault__Append.py, line 20: APPEND_MAX_FILES = 1000, a module constant. The same file sets 5 MB per payload, 100 ids per fetch and 200 entries per list page.
• append(), lines 131 to 134: list every file under bare/append/{token}/pending/; if there are 1,000 or more, return at_capacity without storing anything.
• fast_api/routes/Routes__Vault__Append.py, line 72: at_capacity becomes HTTP 507 Insufficient Storage. The server's own test is test__write__at_capacity_returns_507.
• The storage is not the limit. In production the listing is Storage_FS__S3.folder__files__all, which calls osbot_aws S3.find_files. That follows NextContinuationToken, so it counts past S3's 1,000 keys per call. Raising the constant would work.

The cap is per token, on pending files only. Processed files do not count, and a vault can carry several tokens: configure takes a list of append_anchors, one per token. sgit.ai documents the cap as a bound on abuse ("P12, bounded abuse: ≤ 1000 pending per token", sgit.ai/docs/append-lane-messaging.md): the token is public, so the cap is what limits a flood.

What happens to event 1,001

Checked by running the server's own Service__Vault__Append in memory, with no mocks (the script was in the session's scratch directory):

writes 1-1000: {'ok'} | write 1001: at_capacity      (HTTP 507)
second lane, same vault, write 1: ok
write after draining 100: ok

• The event is lost. The server stores nothing and keeps no count of refusals.
• Nobody notices. assets/telemetry.js sends with fetch(..., keepalive) and swallows every error ("telemetry never surfaces"). The page is not marked as sent for the tab, so the reader's next page in that tab retries only if they come back to the same path.
• The count is approximate. The check and the write are two steps, not one, so writes that arrive at the same moment can each see 999 and land a few over 1,000.
• It recovers on drain. mark-processed moves files out of pending/, and writes succeed again at once.

How close we are

From the application's own figures: 1,543 events from 22 October 13:43 to 24 October 11:06 UTC, about 34 an hour. 407 waited between 23:48 and 11:06, about 36 an hour overnight. At that rate the lane fills about 29 hours after a drain. At ten times the traffic (one link on a busy site) it fills in about 3 hours, and everything after that is lost until someone drains it.

Options

1. Drain on a schedule. The cheapest fix and the one that matters most: every hour or two, the lane never gets near 1,000 at current traffic. This needs a scheduled task (the editor's go) that holds the telemetry vault key.

2. Lanes in parallel (shards). Give the vault N tokens, configure all N anchors, and publish all N tokens in data/telemetry.json. Each page picks one by its session id (hash(sid) mod N), so a session's events stay in one lane. That gives N × 1,000 of headroom with no server change. The drain already handles it: list without an inbox returns every token's entries, each tagged with its inbox, and fetch and mark-processed take the inbox. The cost:

• configure replaces the anchor list, so every call must send all N anchors (threat T10 in the sgit.ai doc), and it needs the SG/Send access token.
• Shards do not help against a flood. Every token is published, so a flooder fills all N, and N × 1,000 is N times the junk.
• The pages need a small change to assets/telemetry.js (dev's file today).

3. On 507, try the next lane. With shards, telemetry.js can retry once on the next token. That spreads load evenly until all lanes are full, and costs one more request at most.

4. Count what is lost. On 507, keep a counter in sessionStorage and send it with the tab's next accepted event (lost: n). The report could then say how many events the cap cost, which today it cannot. No server change.

5. Change the server (a proposal to SG/Send's maintainers, not ours to push): make the cap per vault (max_pending in config.json, default 1,000), and keep a pending count per token instead of listing the folder on every write. Today each write lists up to 1,000 S3 keys, and the cost grows with the cap. A related cost: list without an inbox finds token folders by listing every key under bare/append/, processed files included. Processed files are never deleted unless purged, so every check of the lane gets slower as the lane gets older. Purging processed/ after each drain (the vault keeps its own copy) would bound it.

Proposal

Do 1 first (a schedule, proposed as a card for the editor's go). Do 4 at the same time, so the cost of the cap shows up in the report. Do 2 and 3 when traffic or the measured losses say so: four lanes is 4,000 events, about five days at today's rate. Take 5 to SG/Send's maintainers as an issue. assets/telemetry.js and the drainers belong to dev until board 272 closes.