Brief: moving a heavy vault to pack mode (sgit 0.24.0), step by step

Brief: moving a heavy vault to pack mode (sgit 0.24.0), step by step

Status: accepted. A how-to, tested end to end on the source watch on 23 October 2026 (oq9gflln copied into the pack vault 7r83qic5). Brief by Dinis Cruz (editor); written up by researcher@php.newsroom. The editor's plan is to move the curator's evidence vault and the other heavy data vaults the same way. The measurements, with charts, are on /admin/pack-mode/.

Why

A normal vault stores every object as its own file on the server, and moves them one request at a time. The source watch has 16 commits, 29,861 files and 63,662 objects (1.6 GB), and adds about 5,000 objects a pass. Its pushes took most of an hour, broke against the server's ~30-minute connection limit, and needed retries. A pack vault moves the same content as zips:

The wall. A normal vault this size has outgrown the server's object listing: it times out, so clones fall back to fetching object by object, and pushes that need the listing fail. That will only get worse as a vault grows. A pack vault lists a handful of zips.

Choose: which vaults should move

Move a vault when agents clone it whole, it is pushed often, and it is big. Keep it a normal vault, or keep a normal copy next to it, when any of these is true:

• People browse it. The browser cannot read packs: vault.sgraph.ai, dev.vault.sgraph.ai and vault apps show a pack vault as empty. On this site, /admin/vault/ and the source-watch page link the evidence vault and the watch into the browser, so those links would break.
• Agents read one folder of it (--path, --depth). On a pack vault those download everything, in this version.
• Something reads it with an older client. Every client must be sgit-ai 0.24.0 or later; older ones are refused by name. On this site .github/workflows/synthetic.yml pins 0.23.0, so any job there that clones a pack vault must move to 0.24.0 first.

The steps

Everything below runs in a scratchpad, never in this repository. Nothing prints a key.

0. The editor's go, and one person in charge. A new vault outlives the session (CLAUDE.md), so the editor says go. One agent does the move; every other writer stops writing to the old vault until the cut-over. Writes that land on the old vault after step 3 are not in the new one.

1. Install 0.24.0 apart from the client in use. Don't upgrade sgit under a push that's running. A virtual environment keeps the two apart:

python3 -m venv sgit24 && sgit24/bin/pip install -q sgit-ai==0.24.0
sgit24/bin/sgit version                      # sgit-ai v0.24.0

2. A full, current clone of the old vault. Not a --path or --depth clone: the import needs every object. In it, sgit pull, then sgit status must say "in sync with remote" and "Nothing to commit".

3. The import, with its output kept private. The output carries the new vault key.

mkdir -m 700 secret
sgit24/bin/sgit pack import <full clone> <new folder> > secret/import.out 2>&1
grep -v sgit_ secret/import.out              # the summary without the key line

It's local and nothing is uploaded. On the watch it took 194 s and reported 16 commits, 29,193 trees and 34,453 blobs, 1,614.4 MB down to 1,184.2 MB (x1.36). Note the new Vault ID. The commit ids change; the files, messages and times do not.

4. Seal the new key to the root, and register the vault, before anything else.

node pki-agent.mjs pin sha256:db7f79fdd87e030a
node pki-agent.mjs seal --from-vault <new folder> --name "<new id> vault key" --expires 14 --qr <id>.png --qr-link

In the same commit, add the new ID to your data/vault-register/<you>.json: a vaults row with key_sealed_to_root dated, and a holds row (AGENTS.md, section 6).

5. The first push. It uploads one snapshot zip with the whole history.

cd <new folder> && ../sgit24/bin/sgit push    # "Pushed 16 commit(s), 63662 object(s) uploaded." in 96 s

6. Prove it: a fresh clone must match, file for file. The clone reads the key from stdin, so it never appears in a command line or a log:

L=<new folder>/.sg_vault/local
sgit24/bin/sgit clone - check --base-url "$(cat $L/base_url)" --token "$(cat $L/token)" < $L/vault_key

Then check:

• The clone's log should say Packs fetched (1 pack(s), 63662 objects, 4 request(s) …). If it fetches objects one by one, stop and report.
• diff -rq -x .sg_vault <full clone> check may only show empty folders and __pycache__, which a vault doesn't store.
• sgit24/bin/sgit pack verify in check should say ok.
• sgit history log | wc -l should match in both.

7. Cut over.

• Each writer gets the new key by a QR handshake. The editor seals it to that writer's key, sent as a file, never as text. The writer installs 0.24.0 and clones fresh; old clones cannot pull from the new vault.
• Readers get the new read key. It's in the new clone's output. Replace the old one where it's published: the read_key fields in data/*.json, and the briefs and prompts that name the old vault (for the watch: the curator's prompt and index/promote.json).
• Update the schedule vault (weic8imn) and /team/vaults/.

8. The old vault. It becomes read-only by agreement: no one writes to it again. It stays, as the record and as the browsable copy, until the editor decides. Deleting a vault is irreversible and needs his go.

9. Name a compactor. Every push adds a small zip. Once there are more than about 50, or they hold more than a tenth of the snapshot's bytes, one named agent runs, at a quiet time:

sgit pull && sgit status && sgit pack compact --dry-run && sgit pack compact && sgit pack verify

A clone that was behind then downloads the whole new snapshot on its next pull: 1.2 GB for the watch, about a minute.

What it does not fix

• Local walks. Commit, status, push and pull still walk the working tree. On the watch's 29,861 files, a one-file push took 77 s, and a pull of one commit into a second clone took over 9 minutes, almost all of it local. Pack mode removes the network cost, not the size of the working tree.
• The server's bad hours. On 23 October the server returned 504s on vault listings from 18:00 to 18:35 UTC. Pack vaults make fewer requests, so they meet fewer failures, but not none.

A block for the agent that does the move

Paste this, with the vault named:

Move the vault <id> to pack mode, following admin/content/briefs/2026-10-23__brief__moving-a-vault-to-pack-mode.md
step by step. The editor has said go. Install sgit-ai 0.24.0 in a virtual environment in your scratchpad; do not
upgrade the sgit your other work uses. Take a full clone, check it is in sync and clean. Import into a new folder
with the output in a mode-700 file; never print the key. Seal the new key to the root (sha256:db7f79fdd87e030a) and
add the new vault to your data/vault-register file before pushing. Push, clone fresh from stdin, and check: 4
requests, diff -rq clean but for empty folders and __pycache__, pack verify ok, same commit count. Report the
numbers (import, push and clone seconds; objects; MB before and after). Do not write to the old vault, do not
delete it, and do not hand the new key to anyone but the root; the editor does the handshakes.