Tests under the floor: why the suite is slow, and what is actually wrong
Disposition: live; handed to the next Code Troll as a session goal (Human,
2026-09-14).
Opened by: Human, 2026-09-14: “I’m still curious if there’s something
wrong with the current process or if tests are written improperly. I have a
hunch, no evidence.” This page is the evidence, kept as it arrived, with the
wrong turns left in so nobody re-walks them.
Kept by: Code Troll (CodeTroll_Breccia, then CodeTroll_Amber the same evening, then CodeTroll_Chert). A number carries the
commit it was measured at; a sentence without a number is a claim.
Observed: 2026-09-14, main f23bcaf0 → bc47a68c (Breccia); ab64c742 → 2d88d1a1 (Amber, see “Amber’s sitting”); 196be3f2 → 50870f84 (Chert, see “Chert’s sitting”).
The question
python -m pytest on 24 cores takes ~5.5 minutes; serial it took 47 minutes
on the morning of 2026-09-14. Is that the cost of a large suite, a process
defect, or tests written in a shape that cannot be fast?
Short answer, as of tonight
Three things, in order of size:
- One bench fixture was the wall.
tests/test_ops_command_canary.py’s module fixture cost 206 s on each of two xdist workers and then failed — a PATCH-575 regression, mended by PATCH-593 (this sitting). Roughly 210 s of a ~330 s wall was that one fixture on the critical path. - The remaining tail is a dozen benches, each rebuilding a real-ish world:
test_session_backfill(65 s),test_capability_bundle_spawn(43 s + eight at 6–24 s),test_tool_inventory(ten at 6–10 s),test_ops_store_population_v1(22, 12, 6 s ×4),test_corpus_job_mcp(three at 7–14 s),test_task_context_compiler(three at 8–15 s). Sixty slowest = 1011 s of worker time. These are not badly written; they are integration tests in a unit suite’s default run. - The floor is git-bound now, not sqlite-bound. After PATCH-575 an
ordinary test costs ~0.125 s (was ~0.5 s). Of that, the largest share is
gitsubprocesses on product code paths (a seal issues ~27 git commands), then ordinary sqlite work, then the house/shop pair stores which are still walked fresh (no seam), ~70 s serial per full run.
Nothing found says “tests are written improperly” as a class. The biggest files by count are well shaped (mocked runners, parametrized, in-process).
What was already known (Alloy’s tenure, same day)
- FIND-1875: serial at
30f86071— 4321 passed / 9 failed in 2830 s. Sixty slowest summed to 579 s; the rest a flat ~0.5 s across ~4270 tests. cProfile: 7.8 of 11.4 s insidemigrations.init_dbwalking 34 migrations per fresh store — 2308 sqlite stores per run, 1346 of them to v35. - PATCH-575
324f0e81: rootconftest.pywalks one pristine v35 store per process and copies it onto every fresh path (_seed_fresh_storeseam insidemigrations.init_db). PATCH-576/578/580/581:-n autopinned inpytest.ini, worktree root fenced, pytest scratch kept by pytest’skeep=3. - Subprocess launches are not the monster:
python corpus.py --help0.15 s,import tools.corpus_ops0.105 s,git init+config+commit 0.21 s (this host, idle). The suite has 405subprocess.*call sites in 72 test files, 335 spawningcorpus.py, ~240 CLI-helper call sites (self._run,self._cli,run_ops).
What this sitting measured
Shape of the suite (main f23bcaf0)
150 test files; 3392 def test_ (4327 collected); 351 unittest.TestCase
classes; 12 pytest fixtures; 681 mock.patch sites; 874
TemporaryDirectory/tmp_path; 30 git init sites in 34 files; 13
time.sleep. Largest by count: test_job_completion_wake.py 547,
test_ops_store_split_v1.py 274, test_capability_bundles.py 187,
test_capability_bundle_spawn.py 130, test_corpus_ops.py 129,
test_print_session_open.py 127, test_ops_job.py 124,
test_ops_store_split_seam.py 116.
The wall was one fixture (--durations=60, -n auto, main f23bcaf0)
206.32s setup tests/test_ops_command_canary.py::test_cross_owner_terminal_and_real_land_receipts_are_visible
206.05s setup tests/test_ops_command_canary.py::test_every_real_leaf_is_invoked_on_both_grounds
64.82s call tests/test_session_backfill.py::JobRoleBackfillTests::test_grok_apply_archives_binds_and_second_dry_run_is_idempotent
42.77s call tests/test_capability_bundle_spawn.py::ProductionSpawnTests::test_controller_verifier_setup_containment_cleanup_and_bind_failures_contact_no_provider
23.99s call tests/test_capability_bundle_spawn.py::ProductionSpawnTests::test_codex_eyes_failed_checks_reach_review_once_but_cannot_accept_pass
22.15s call tests/test_ops_store_population_v1.py::test_read_canary_names_the_command_a_damaged_pair_misreads
16.16s call tests/test_capability_bundle_spawn.py::ProductionSpawnTests::test_codex_eyes_missing_receipt_and_mutation_still_fail_closed
15.12s call tests/test_task_context_compiler.py::test_replay_req245_mandatory_closure_predicts_the_real_amend_gap
14.21s call tests/test_corpus_job_mcp.py::PersistedLifecycleParityTests::test_all_enabled_hosts_persist_existing_family_provider_pairs
12.20s call tests/test_ops_store_population_v1.py::test_read_canary_over_a_populated_pair_matches_the_mouth
12.12s call tests/test_capability_bundle_spawn.py::ProductionSpawnTests::test_review_pending_candidate_holds_scope_until_request_moves
11.06s call tests/test_patch_backlog.py::LandingTests::test_cherry_pick_on_main_binds_as_landed_history_after_the_ref_goes
11.00s call tests/test_corpus_job_mcp.py::PersistedLifecycleParityTests::test_direct_grok_and_codex_ingress_persist_identical_lifecycle_shapes
10.64s call tests/test_capability_bundle_spawn.py::ProductionSpawnTests::test_codex_eyes_large_evidence_stays_off_argv
9.55s call tests/test_tool_inventory.py::RealTreeRegenerationTests::test_database_untouched_read_only
9.55s call tests/test_ops_command_canary.py::test_dispatch_evidence_is_deterministic_across_temporary_roots
9.18s call tests/test_scout_semantic_inventory.py::test_live_pilot_has_no_mutual_cycles_and_atomic_req_change_neighborhood
(… ten test_tool_inventory tests at 6.4–9.6 s; eight more test_capability_bundle_spawn at 6.5–9.1 s;
test_console_encoding 7.8 s; test_corpus_story ten_thousand_trace_events 7.8 s; test_retire_session cross_host 7.8 s …)
xdist cannot share a module-scoped fixture across processes: a module fixture
runs once per worker that draws a test from that module. That is why the
canary appears twice. --dist loadfile (all tests of one file on one worker)
would make every module fixture run once; untested here — try it and measure.
The canary red: a PATCH-575 regression (FIND-1888 → PATCH-593 bc47a68c)
Bisect in throwaway worktrees, tests/test_ops_command_canary.py -n0:
| commit | result |
|---|---|
324f0e81^ (before the seam) |
36 passed in 214 s |
324f0e81 (PATCH-575) |
32 passed, 4 errors in 21.7 s |
2bf346bd (shared checkout) |
32 passed, 4 errors |
bc47a68c (PATCH-593) |
37 passed in 107 s (goblin, busy host) |
Mechanism, measured with a one-shot probe: the fixture full_report wraps
sqlite3.connect and asserts every path is under ops-command-canary- or
:memory: before counting. The first fresh init_db in a worker builds the
process’s pristine store at %TEMP%\corpus-pristine-<x>\pristine.sqlite3
(conftest._pristine_store); the guard raises inside the wrapper,
canary.run() swallows it into report["failures"], count stays 0, wall
0.0 s. (The 206 s in the durations run is the same bench under 24-worker load
on a worker where the pristine store already existed, so the run went the
distance and the count was still short — not re-measured; claim.)
Probes, same commit: counting wrapper without the guard — 3530 connects, 83 s,
status FAIL, 48 of 58 leaves difference. Seam disabled inside the run —
3529 connects, 80 s, still 48 differences. So (a) the seam neither speeds
nor slows the bench: leaf work dominates, not store creation; (b) the 48
differences are by design — tests/test_ops_command_canary.py:80 asserts
status == "FAIL" while the store-split ring is unrewired (NEWS-414).
Disproven on the way: “the pristine store is walked under the real clock
while the bench pins now, so seeded rows carry foreign stamps.” Disabling
the seam changed nothing. Left here so nobody re-walks it.
PATCH-593’s cut: deterministic_runtime pins migrations._seed_fresh_store
to a locally declared no-op (_walk_fresh_store) so the bench walks its own
stores, as its docstring already claimed. The guard is unchanged. Goblin
concern on TRACE-2244: the shipped no-op has no importable name —
conftest.py rebinds the module attribute at import, so both this PATCH and
tests/test_conftest_pristine_store.py:56 re-declare the shape locally. A
handle captured at definition in migrations.py (e.g.
_WALK_FRESH_STORE = _seed_fresh_store) would let both pin the real function.
The floor after PATCH-575 (tests/test_corpus_ops.py, serial, f23bcaf0)
129 tests in 16.1 s = 0.125 s/test (pre-575: ~0.5 s). cProfile of the file:
| where | cumulative | note |
|---|---|---|
subprocess.run |
4.7 s (186 runs) + reader threads | 137 via capability_bundles._run_git; 117 through _git_ok; 75 identity_git |
five ReleasedHandoffBriefTests seal tests |
7.3 s | job.end_job → sign_out_agent → _release_writer_lease_after_sign_out → release_writer_worktree: ~27 git calls per seal |
sqlite3.Connection.execute |
2.3 s (15 034 calls) | ordinary work |
store_split_v1.initialize_store |
1.5 s (10 calls, 147 ms each) | house/shop pair stores still walked fresh; the seam covers only the monolith. Alloy counted 469 pairs per run → ~70 s serial |
migrations.init_db real walk |
0.04 s (1 call) | the seam holds: one walk per process |
The 0.5 s serial floor has not been re-measured across the whole suite since PATCH-575 (claim: it fell ~4×; the whole-suite serial number is owed).
Host facts that bite tests
- System gitconfig
core.autocrlf=true(‹redacted: local coordinate›); the repo pinsfalse. Tempgit initrepos inherit the system value. PATCH-582 had to pinautocrlf=falseintest_retire_session.pyfixtures; 34 test files do agit init. Any that push CRLF bytes through a blob will lie on another host. Untested claim; a sweep, not yet a hole. - Only this process’s cwd is knowable to a sweep; a parent shell inside a Hole tree still locks it on Windows (FIND-1885, PATCH-587).
Reds on main tonight (after PATCH-593), none of them speed
test_claude_provider::test_root_claude_md_is_lightweight_agents_doorplate
(Human’s file; Human said keep the test); test_native_identity (v34 vs v35
assertion); test_capability_bundle_spawn::test_historical_recorded_root…
(the broom writes _cleanup_last.json into an external root);
test_ops_patch_job (sealed-SHA claim); test_ops_reviewer_preflight (three
tools missing OPS MAP); test_ops_find tendril flag wording; two
test_provider_dispatch_reconcile reservation-key field sets. -n-only
flakes: test_corpus_controller…digests_at_job_end,
test_ops_news_successor_req274…[file]. FIND-1875 lists causes.
Amber’s sitting (evening 2026-09-14, main ab64c742 → b4482a2e →)
Correction (Chert, FIND-1895):
-n autostarts 12 workers on this host (xdist counts physical cores), not 24. Every “24 workers” in this section and the next that came from a bare or-n autorun was 12. Spawn censuses that saw “13 of 24” files were complete (12 workers + controller), not lower bounds.-n 12vs-n autocompared 12 with 12.
Human’s word tonight: “less than 5 minutes” → “less than a minute, instant” → “less time would be spent just waiting for tests.” The floor decision the bench-marker PROP was waiting on is made by that word (FIND-1890). Human also asked whether tests are written inefficiently, why the suite chokes, and whether anyone screened what is necessary — and passed on an advisor’s read (kept in FIND-1890’s spirit: selection is not optimization; the census is a placement review, not a deletion exercise; test each guarantee at the smallest boundary that can demonstrate it).
The numbers (this host: Ryzen 9 9900X, 12 cores / 24 threads)
| run | commit | wall | result | file |
|---|---|---|---|---|
serial -n0 --durations=0 |
ab64c742 |
1655 s (27:35) | 4345 passed / 9 failed | durations/serial-ab64c742.txt |
shipped -n auto --durations=0 |
b4482a2e |
340.9 s | 4347 passed / 8 failed | durations/parallel-n24-b4482a2e.txt |
| pair-store files serial, after PATCH-594 | b4482a2e |
217.9 s (was 342 s) | 446 passed | durations/pairfiles-serial-b4482a2e.txt |
--co |
b4482a2e |
0.68 s | 4358 collected | — |
Two facts the earlier page did not have:
- Under 24 workers every test costs ~2× its serial time (3208 s of worker
time for 1649 s of serial work). Twelve physical cores, 24 threads, and
git/sqlite/subprocess all contending for the same disk.
-n 12for the fast tier is untested and worth one run. - The wall is 2.5× the contention-adjusted ideal (340 s vs 3208/24 =
134 s): a schedule tail. The canary’s module fixture still ran on two
workers (92 s each) because
--dist loaddoes not keep a file together.
durations/sum_durations.py <file> prints per-file worker time and what each
bench threshold would buy; durations/draw_line.py <file> draws the line.
FIND-1889 → PATCH-594 b4482a2e (imp-verified, TRACE-2247)
store_split_v1.initialize_store ran the pair schema through executescript
in autocommit under journal_mode=DELETE: one journal write + fsync per
CREATE, ~160 ms per store. Inside one BEGIN … COMMIT: 9 ms. sqlite_master
identical; journal mode untouched; failure path untouched; ratchet traces the
statements. Alloy’s “synchronous=OFF gives 17×” was this cost seen from the
other side — no durability premise had to move. Pair-store test files:
342 s → 217 s serial.
The seal’s git calls (idea 5): NULL for the suite’s sake
Instrumented _run_git across the five ReleasedHandoffBrief seals: 24
_run_git calls, ~650 ms per seal, plus ~10–15 _identity_git reads on the
other rail (not captured). Composition per seal: ls-files --cached --others
×3, ls-files --deleted ×2, rev-parse --is-inside-work-tree ×2, worktree add ×2, worktree remove ×2, rev-parse ^{tree} ×2, and one each of
show-toplevel, HEAD, diff --name-status (×2 forms), verify main,
ls-files --cached, add -A, commit, update-ref, worktree list. The
writer’s ls-files listing is read four times inside the one held lock
(_expected_handoff_digest, _copy_candidate, _prune_paths_dropped_by_writer,
candidate_identity) — that is one unchanged phase and could be one read.
Worth ~100 ms per seal; ~130 seal sites in tests → ~15 s serial, under a
second parallel. Not worth a product-code cut for the suite; may be worth one
for live seals later. The repeated reads across worktree-add and commit are
deliberate and stay.
Idea 7 (a named handle for the seam no-op): NULL
A return None has no behaviour to drift from; two local five-line
declarations are not a burden anyone pays.
FIND-1890 → PATCH-595 (mechanism) + PATCH-596 (marks)
The line is bench-line-b4482a2e.md: 12 whole files and 33 single tests. The
fast tier keeps ~429 s of serial worker time → 18 s ideal, ~36 s at the
measured contention, plus start-up; max remaining test 1.98 s. Mechanism: a
bare python -m pytest deselects bench; --bench, a named path, -m or
-k runs exactly what was named (a verify can never pass on zero tests);
main_health runs --bench. Both landed and imp-verified: PATCH-595 e406d463 (TRACE-2250), PATCH-596
2d88d1a1 (TRACE-2251). Measured below.
The fast tier, measured (main 2d88d1a1, PATCH-595 + PATCH-596 landed and imp-verified)
| run | wall | result |
|---|---|---|
bare python -m pytest (-n auto = 12, not 24; FIND-1895) |
84 s | 3629 passed / 7 known reds / 724 deselected |
-n 12 |
88 s | +1 flake (test_custodied_process…exact_environment) |
--dist worksteal |
85.5 s | same |
--durations=0 at 24 |
94 s | worker time 989 s for 429 s of serial work |
| xdist fixed overhead (1 file, 24 workers) | ~5 s | vs 2.3 s serial for the same file |
340 s → 84 s. Not under a minute yet, and the reason is now measured rather
than guessed (durations/fast-*.txt, probes/):
The floor is process-spawn-bound, not core-bound. probes/microbench.py
idle vs under a 24-worker run: pure Python 17.6 → 20–25 ms (hyperthreading,
expected); git --version 10 → 47–59 ms; python -c pass 16 → 27–32 ms;
git init+add+commit 78 → 290–316 ms; sqlite WAL create+commit+close 16 →
40–59 ms. -n 12 being no faster than -n 24 says the same thing. Windows
process creation and the filesystem are the contended resource; adding cores
adds contention.
How many spawns: probes/spawn_probe.py (13 files = 12 workers + controller: complete, FIND-1895; originally read as “13 of 24 workers reported — a
lower bound): ≥ 9 252 child processes per fast run, ≥ 8 895 of them git,
≥ 1 649 s of child wall (python children 1 145 s — long-lived stubs and
corpus.py runs; git ~500 s). At ~55 ms per git spawn under load that is
~90 % of the 989 s of worker time. probes/spawn_probe2.py attributes them:
- From test fixtures directly (no
tools/frame):git config629,add565,commit517,init391,rev-parse274,worktree118 — roughly 400 hand-built fixture repositories per fast run at 5–6 spawns each. 34 files do their owngit init. A per-process pristine repository copied withshutil.copytree(the PATCH-575 shape, for git) would take ~2 000 spawns out; but the fixtures are hand-rolled per file, so this is the placement review, not a one-line seam. - From product code:
capability_bundles._run_git~1 400 +_identity_git~660 +hash_object185;patch_git_images._git~1 300 (ls-tree294,show269,rev-parse176,for-each-ref169,ls-files152 …);main_health._git180;git_backup._run~190;git_checkpoint._git_bytes~180;eol_drift._hash_paths90. - By test file:
test_ops_patch_receipt.py1 941 (22 per test),test_source_custody.py830,test_ops_job.py766,test_retire_session.py741,test_ops_patch_job.py690,test_git_backup.py624,test_git_checkpoint.py526,test_corpus_controller.py469,test_ops_schema.py425.
The extra red at -n 12 (test_custodied_process…shell_false_binary_capture_and_exact_environment)
did not recur at 24; a flake under load. Chert’s scout: not reproduced in 7 tries at 94ddbd3a; the
test mocks its process, so a spawn-load race is implausible. Capture the traceback if it recurs.
Word sent: Architect MSG-3838 (FIND-1890 overtakes MSG-3836; the seam and
the spawn-bound premise), Orchestrator MSG-3839 (default run changed shape;
main_health runs --bench).
Next holes, in order (for whoever sits next)
Status after Chert’s sitting: 1 partly done and re-aimed, 2 split into named holes, 3 begun (four files), 4 scouted with a working prototype. See “Chert’s sitting”.
tests/test_ops_patch_receipt.py: 22 git spawns per test × 90 tests. Read what each test needs from git and what its fixture builds; this one file is a fifth of the fast tier’s spawns.patch_git_images._git~1 300 calls per run — count the distinct facts one image needs against the calls it makes (the seal’s method, FIND-1889’s page). Product code; imp with care.- A pristine fixture repository per process for the ~400 fixture repos:
tests/house_ground.pyis the natural home; each file’s hand-rolledgit initbecomes a copy. Placement review per file, not a global seam. - In-process CLI dispatch for the 197
python corpus.pyspawns (41 s of child wall, more under load): the advisor’s middle boundary — the same callable entry point serving both the executable and tests. - The census (placement review): per file, what it pins, whether that is still on the map, what wrong behaviour it alone would catch, and the smallest boundary that still demonstrates it.
Under a minute is reachable from 84 s by items 1–3 alone (each ~100 ms of worker time per spawn removed under load); “instant” needs 4 and 5.
Answers to Human’s questions, as evidence stands
- Changing tests or how we run them? Mostly the floor under them (575, 594) and one fixture (593); the tier split changes how we run, not what exists.
- Written inefficiently? Not as a class. The one expensive shape choice is
driving the real CLI as a child process: 405
subprocesscall sites in 72 files, 335 spawningcorpus.py— an opportunity count, not a savings number; nobody has measured what converting them would give. - Why choked? One regressed fixture (fixed), ~60 integration benches in the default run, a 0.125 s floor from real git/sqlite/subprocess, and 2× contention at 24 workers.
- Screened for necessity? No. 4358 tests, 351
TestCaseclasses, 150 files, no census. The next arc: a placement review per file — what it pins, whether that is still on the map, what wrong behaviour it would catch that the rest would miss, and the smallest boundary that still demonstrates it.
Chert’s sitting (2026-09-14, main 196be3f2 → 50870f84)
Every number below says who observed it. “Scout” and “goblin” numbers were reported to Troll; the ones marked checked Troll reran or recounted.
FIND-1891: the receipt file (hole 1). What was measured, and what was wrong
Serial at 196be3f2 (Troll, probes/spawn_probe2.py in-process): 97 tests, 56.4 s,
1 941 git spawns. Fixture 571 (init 90, config 271, add/commit 212); the rest
product frames.
- PATCH-597
fdc34c27(imp TRACE-2254): the fixture’s threegit configspawns became a write to.git/config. −3 spawns/test; no measurable wall (goblin: 52.0 → 51.5 s serial on a loaded host, within noise). - PATCH-598
dea5b786(imp TRACE-2256):patch_git_images.commit_path_imagesbatched: onels-tree -zper side and onecat-file --batch, instead ofls-tree+showper path per side. FIND-1891 blamed the wrong reader. For this file the goblin measured 1 941 → 1 901 spawns, 56.7 → 54.7 s: its fixtures declare 1–2 paths. The gain is on live receipts with many paths. Checked (Troll,probes/commit_images_differential.py): 678 path sets, 0 divergent from the old reader (folders with files under them, globs, case,./a,d//g,:a, NTFS-invalid names); a deleted loose blob raises in both; spawns old → new: 1 path 6 → 5, 4 paths 17 → 5, 27 absent paths 56 → 4, 25 mixed paths with odd spellings 83 → 29 (odd spellings fall back per path, by design). The imp separately ran 2 085 sets, 0 divergent. The added code is 174 lines; a plaincat-file --batchofref:pathis not simpler in truth: it answers “missing” for both a bad ref and an absent path (checked). - The real cost of this file is
head_blob(tools/ops_patch_receipt.py:161):path_imagesreads HEAD per path (:343) andunified_diffreads it again for every path on disk (:290). Checked by reading. Next Hole; product code.
FIND-1893 → PATCH-599 50870f84: build a fixture repo once per process (hole 3)
tests/fixture_repos.py::copy_built_repo(dest, key, build): the class’s own git
steps run once per process into a cache, then each test gets a copytree.
Adopted by test_git_backup, test_git_checkpoint, test_source_custody
(StandingSourceCustodyTests only), test_declared_ground.
Imp-verified, TRACE-2257: builders equal the parent’s setUp line by line; parent-built
and copied repos equal on trees, refs, tags, local config, index, status and bytes;
passes under -n auto and -n 4; caches removed at exit; failed builds cache nothing.
Known limit (imp’s concern, Troll reproduced): the non-Windows fallback runs
update-index -q --refresh, which exits 1 on a conflicted index, so a builder that
leaves a merge conflict would raise there. --unmerged does not fix it (also exit 1,
checked). No builder leaves one; NULL until one does.
| file | git spawns before → after (goblin, probe) | serial wall back-to-back (goblin, loaded host) |
|---|---|---|
test_git_backup.py |
624 → 318 | 21.3 → 9.4 s |
test_git_checkpoint.py |
526 → 411 | 10.1 → 9.4 s |
test_source_custody.py |
830 → 670 | 27.7 → 23.9 s |
test_declared_ground.py |
101 → 57 | 3.4 → 3.4 s |
Checked: each saving is exactly (tests in the class − 1) × spawns per build: 306 = 34×9, 115 = 23×5, 160 = 32×5, 44 = 11×4.
The scout’s copy numbers were a load artefact. Scout, loaded host: init + config +
commit 928 ms, copytree 356 ms, copytree without hooks/*.sample 23 ms. Troll,
idle host, medians of 20 (probes/init_vs_copy_microbench.py): git --version
10.7 ms, git init 26.3, init --template= 19.7, copytree 10.3, without hooks
5.1. The win is spawns avoided, not copy speed.
A trap the goblin found, and Troll reproduced. A copied file gets a new
Windows creation time, which Git for Windows reads as st_ctime. With the cache
aged 3 s, a copy made git diff-files report every tracked file modified while
git status (which refreshes the index) said clean. The helper carries the
creation time across with SetFileTime (about 60 lines of ctypes), falling back
to one update-index --refresh elsewhere. Checked: carry → 0 dirty; no carry
→ 5 of 5 dirty; no carry + core.trustctime=false → 0 dirty. The four adopters
pass with carry and refresh both off (110 passed): the guard is for later
adopters that read the index with plumbing. core.trustctime=false in the built
repo is the one-line alternative if the ctypes ever needs nursing.
Not adopted, for a reason: test_update_source_repos.py:24 (a clone’s config holds
an absolute remote path); ReviewPendingSectionJobStartTests (different build, 2
tests); whole-file bench files. Next adopters (scout census from source, not
probe-measured): test_ops_schema.py:767 (3 × 25), test_ops_patch.py
seal/retained classes (4 × 14), test_corpus_controller.py:186 (3 × 14).
Scout: in-process CLI dispatch (hole 4)
Scout, probes/cli_spawn_probe.py + cli_spawn_agg.py, fast tier at fdc34c27:
203 corpus.py child spawns (the “335” above counts source lines that
mention the string, not spawns — scout’s claim, not rechecked); child wall
32.1 s serial, 56.0 s under 24 workers; ~184 of them in ten ops/story files
(test_ops_req_get_render 32, test_ops_find 23, test_ops_patch 21,
test_ops_mail 20, test_ops_tendril 19, test_ops_prop 18, …).
corpus.main(argv) -> int exists (checked: corpus.py:195) and tests already
call it. Prototype probes/corpus_inproc_prototype.py on copies of four files:
same outcomes (52 passed, 1 known red), 90 → 1 spawns, 20.9 / 21.7 s → 6.1 / 4.2 s
serial (checked: the scout’s timing file matches its report). No product
change needed. Keep true process tests: cp1252 stdin (test_ops_find.py:397),
strict console (test_corpus_entrypoint.py:55), hooks, MCP stdio, supervisors.
Hazard: cached office tables in tools/print_session_open.py (_OFFICES_DATA,
_MODE_TABLES) if session/role tests convert. Worker-time saving ~25–45 s
per fast run is inferred; wall at -n auto likely a few seconds. The scout’s
serial fast tier took 920 s under load against Amber’s ~430 s; treat its
absolute numbers as loaded.
Scout: product git calls beyond PATCH-598 (hole 2, split)
Ranked from one probed -n auto run (13 files = 12 workers + controller: complete).
Troll checked by reading the first three.
accept_patch_commit_verification(tools/corpus_ops_v1/domains/house/patch.py:787-788) validates the same first parent incommit_path_imagesand again incommit_changed_paths: 2 spawns per live acceptance. PATCH, small.head_blobread twice per path inbuild_receipt(above). PATCH; the suite’s biggest single-file consumer.candidate_identity(tools/capability_bundles.py:1601-1602):rev-parse HEADandrev-parse HEAD^{tree}can be onegit rev-parse HEAD HEAD^{tree}. 1 spawn per identity read on the seal rail. PATCH, trivial.git_checkpoint._unresolved_paths(tools/git_checkpoint.py:168-193): the--diff-filter=Udiff and the plain diff may be one--name-statuscall. Scout tested a real conflict; not rechecked.- Seal (
capability_bundles._run_git,_identity_git): NULL for the suite, as Amber found. main_health._git: three distinct facts. NULL.
A fresh run: what PATCH-597/598/599 bought (Chert, A/B)
A = 94ddbd3a (before PATCH-597), B = c8500ae7; between them only those three PATCHes change code
(and CLAUDE.md, FIND-1892, which fixed the doorplate red). Two detached measurement worktrees under
.work/chert/ab/, same host, back to back; worktrees skip 8 live-ground tests. Spawns counted by
probes/spawn_census.py, a pytest plugin that reports from every worker at pytest_sessionfinish.
| A | B | |
|---|---|---|
| git spawns | 8 865 | 8 266 |
| git spawns without the two new test files | 8 865 | 7 914 (−951, −10.7 %) |
| python children | 351 | 352 |
| wall at 12 workers (s) | 81.8, 86.1, 101.2 | 84.5, 79.2, 108.1 |
Per file (git): receipt 1 941 → 1 631, git_backup 624 → 318, source_custody 830 → 700, git_checkpoint
526 → 416, ops_patch 282 → 231, declared_ground 101 → 57; no other file moved. Red set identical but for
the doorplate fix and two one-off known -n flakes. The spawn cut is real and inside the wall noise:
host load drifted 20 s between runs of the same tree.
FIND-1895 → PATCH-601 82240e67: -n logical (imp-verified TRACE-2260)
-n auto = 12 here (psutil.cpu_count(logical=False)); -n logical = 24. Measured in B, alternating:
| workers | fast tier wall (s) | median |
|---|---|---|
| 24 | 77, 58, 58, 60, 78 | 60 |
| 12 | 80, 73, 88, 81, 89 | 81 |
| 16 | 99, 101 (last; host load rising) | — |
| 6 | 102 | — |
--bench: 24 → 296 s, 12 → 330 s (a goblin likely loaded the 12 run; the gap is an upper bound).
Measured, not explained. The imp’s own bare run on main at 24 took 104 s while the PATCH-600 goblin was
very likely running tests: not a measurement of the change. A clean bare run on main at -n logical
is still owed. Word: Orchestrator MSG-3843, Architect MSG-3844 (corrects MSG-3838’s “cores only add
contention”).
Late sitting: FIND-1894, FIND-1896, FIND-1897, one abandon (Chert)
- PATCH-600
cea85cbf(FIND-1894 item 1; verified TRACE-2267):patch_git_images.ref_path_blobsis public;build_receipt’s write lane reads HEAD blobs once forunified_diffandpath_images; the drift check stops reading HEAD preimages it discarded. Onebuild_receipt, 1 modified path: 9 → 7 spawns; 5 paths: 37 → 19 (goblin, confirmed by two imps with counting runners). Imp_Flint’s 32-case differential found two intended changes only (messages on a repo with no commit; drift check on an orphan branch no longer raises from a discarded read). First seat TRACE-2263 recorded FAILED because the verify command includedtests/test_ops_patch_job.py, which carried a stale red. - PATCH-604
392d76ad(FIND-1875; verified TRACE-2266): that stale red. The store now demandscommit_retentionbefore it reaches the sealed-candidate refusal the test pins; the test supplies real retained-candidate evidence. Mutation: with the sealed check defeated, the test fails. - PATCH-602 abandoned (TRACE-2265): a process-global memo for
_verified_commit_parent. The goblin’s work held (1 990 → 1 658 git spawns across the receipt and ops_patch files at-n0), but the memo is only an existence check while every caller follows it with a git read that fails for a missing commit; nothing enforces that. If the duplicate parent check is ever worth cutting: resolve once at the top of the operation and pass it down. - PATCH-603
e9695110(FIND-1896; verified TRACE-2268): the Hole sweep’s finalrmtreeclears git’s read-only bit and retries; landing PATCH-600 had died on a goblin’s scratch repo. - FIND-1897, no PATCH yet:
_tracked_path_spellinguses:(icase), which glob-matches (declaredg[1].txtbeside trackedg1.txtis refused as a casing mismatch; reproduced).:(icase,literal). Also PATCH-600’s comment inbuild_receiptclaims a HEAD-move window closed; HEAD is still named byls-tree,read-treeanddiff.
Whole truth on main e9695110 at -n logical: python -m pytest --bench 295 s, 4 389 passed, 7 failed:
the five known reds, test_capability_bundle_spawn…historical_recorded_root (known bench red), and
test_tool_inventory.py::CliTests::test_cli_navigable_unenrolled_entrypoint_candidate_filter —
TypeError: 'NoneType' object is not iterable in tools/scout_semantic_inventory.py::dispatch_returns
while scanning the repo; passes alone at the same HEAD. Unexplained; likely a file changing under the scan
mid-run. Capture the file it was parsing if it recurs. Fast tier on main at -n logical was not measured
cleanly this sitting (the one bare run, 104 s, overlapped a goblin).
Next holes after Chert (in order)
- In-process CLI helper for the ten
ops/storyfiles (tests only; scout prototype inprobes/). - More
copy_built_repoadopters:test_ops_schema.py:767,test_ops_patch.pyseal/retained classes,test_corpus_controller.py:186(one placement read per class). - FIND-1897 (
:(icase,literal)and the overclaiming comment), one PATCH ontools/ops_patch_receipt.py. candidate_identity’s tworev-parsecalls as one;git_checkpoint’s two diffs (scout, not rechecked).- A clean fast-tier measurement on main at
-n logical, then the census.
Never put a test file with a known red in a PATCH’s verify command (PATCH-600 sat an hour behind one).
Flitch’s sitting (2026-09-14/15, main 2dbb29fc → ac6e3462)
Every PATCH below was written by a goblin in a Hole tree and checked by a separate imp; the TRACE is the seat.
The floor, measured by Troll on this host
| main | fast tier | wall (pytest) |
|---|---|---|
2dbb29fc (arrival) |
3667 passed, 5 failed | 65.07 s |
bbb1960c (after 605–610) |
3683 passed, 0 failed, then the NEWS flake in 3 of 6 runs | 55.9–74.4 s |
ac6e3462 (after 611–612) |
3683 passed, 0 failed, 3 of 3 runs | 60.2 / 73.0 / 75.4 s |
The wall is noisy (another session was live on the host: library/CODEX.md dirty). The floor is green. It is not reliably under 60 s.
What landed
- PATCH-605
2c31907a(FIND-1897, TRACE-2274): receipt reads name declared paths literally.:(icase,literal)for the casing check;GIT_LITERAL_PATHSPECS=1on the worktree and commit diffs and on intent-to-add (the argv flag broketest_the_receipt_module_runner_governs_the_head_read, which pinsargs[0]). Troll’s widening:git diff HEAD -- g[1].txtalso listed a modifiedg1.txt. The imp split the three fixes and showed each one is needed. - PATCH-606
64db38fc: WORKING_SEAMS names the$CLAUDE_SCRATCHPADtrap (MSG-3841). - PATCH-607
7c470d88(FIND-1892 §4, TRACE-2273): three stale reds.ops.SCHEMA_VERSION; TRACE-001 as the unsupported tendril kind (PATCH joined the kinds);notification_dispositionin the reservation keys (PATCH-572). - PATCH-609
3404f686(TRACE-2278): OPS MAP forhole_locate,ops_command_canary,patch_git_images. The ASTs are identical to the parent. - PATCH-608
3f810f27(TRACE-2275), PATCH-610bbb1960c(TRACE-2279): hole 4 begun.tests/corpus_cli.run_corpuscallscorpus.mainin-process (FIND-1892 §6 rule in its docstring, with the hazards). Six files adopted. Serial-n0, imp-timed: req-get/mail/tendril 13.8 → 3.9 s; find/patch/prop 23 → 15 s. The two cp1252 stdin tests still spawn, and a comment says why. The in-process route skipsemit’s byte branch; every remaining spawning test still covers it. - PATCH-611
bfe2edb0(FIND-1898, TRACE-2281), PATCH-612ac6e3462(TRACE-2283): the “known-nflake” was a clock-second coincidence.differential._Normalizernumbers timestamps by first appearance. A fixture stamped by the real clock could share a second with one observation’s write, on one side only. The fixtures are now built under a pinnedclock.utc_now. Imps forced the coincidence: non-equivalent at the parent, equivalent at HEAD, for both files. WORKING_SEAMS now names one flake (controller).
NULLs, with reasons
tools/hole_tree.py:302,307git add/commit -- declaredstill glob. A modified undeclared sibling is refused first by the extra-dirty check; an untracked one would be caught by the imp’s changed-paths check;git ls-files | grep '[][*?]'counts 0. Cheap to add--literal-pathspecsif a glob-named file ever appears.GIT_LITERAL_PATHSPECSset in a caller’s environment disables the:(icase,literal)magic, and the casing check passes silently. This predates PATCH-605, and nothing sets it.- FIND-1898’s residual: a route whose own writes straddle a second can still number differently. The window is milliseconds.
Unexplained, not chased
--bench red test_tool_inventory…navigable_unenrolled_entrypoint_candidate_filter: the traceback in .work/chert/bench-main-e9695110.txt (swept soon) is TypeError: 'NoneType' object is not iterable inside ast.iter_child_nodes, reached from scout_semantic_inventory.dispatch_returns in a tool_inventory.py child scanning tools/. It passes alone. Guess (unconfirmed): a file in tools/ is mid-write by a parallel test.
Next holes after Flitch (in order)
- More
run_corpusadopters. Scout census (probes/cli_spawn_agg.py .work/chert/scout-cli/raw/fast) had 203corpus.pyspawns; roughly 90 are now gone. Next by mention count:test_print_session_open(mind_OFFICES_DATA/_MODE_TABLEScaches),test_print_spawn_brief,test_corpus_cli_map,test_ops_job,test_corpus_wayfinding,test_ops_core_lists. Keep spawns for stdin, env, cwd,os.chdir(source_custody --land), and--*-file -. - More
copy_built_repoadopters (Chert’s list stands). - The same census shows 47
python -cchildren with 1685 s of child wall: long-lived helpers, not rechecked. Worth one look at what they wait on. - A
--benchrun on main, andtools/main_health.pytoward FIND-1892 witness 2. candidate_identity/git_checkpoint(scout, still not rechecked).
Git has no house caller (FIND-1900, MSG-3847 to Architect). Human’s word, late in the sitting: “collapse git into one caller… it’s time.” The inventory lists about 20 wrappers and four ways of making pathspecs literal, and the rest glob. It also names the repeated-fact Holes that can sail now (hole 5 above among them; candidate_identity is now rechecked). The runner’s floor waits on Architect. Before cutting another git call site, read FIND-1900.
The outer bench on the git caller (archive/outer-bench-git-caller-2026-09-15/, two files, read the revision second; to Architect as MSG-3848/3849). The revision is the one to act on:
- Put the shared caller underneath
patch_git_images’ existingGitRunnerand batches, not beside them. Keeprun_childgeneric and give git a bytes route through it. - Strip inherited
GIT_*first, then apply the operation’s own temporary index, object store and identity. Corpus identity is an explicit choice, not a default. - Selection stays command-specific:
ls-treedoes not glob, and*.mdand:(icase,literal)are named requests. - Tests keep sequencing assertions but get logical commands; runner tests own the flags.
- Order: regression fixtures, then the adapter, receipt and safety consumers, mutations, utilities; repeated facts run separately throughout.
For the speed goal, say it plainly: consolidation is a correctness project. Counted honestly, the six repeated facts save about five launches per operation. Receipt HEAD reads save none (they gain consistency; PATCH-600 already made the blob read once). Getting under 60 s still needs what this sitting did not do: a --durations and per-worker picture of the fast tier on a quiet host, then asking whether the slow tests should exist before choosing a remedy. Troll’s own arrival mistake: following inherited hole lists instead of measuring where the wall goes.
The bench also corrected the inventory: a capture_output=True, text=True call decodes the output even if the caller only reads the return code (house/offices/archivist/library_catalog.py:766).
Driving a Hole from Troll’s own session: --checkout moves the host’s cwd into the tree, and --land from there fails its sweep with Permission denied. Land from the shared checkout, and tell goblins to do the same.
Ideas, in the Troll’s order (eliminate → simplify → clarify → repair → automate)
- Measure first, whole suite, serial,
--durations=0to a file at the current HEAD. One number nobody has since PATCH-575. Then the same with-n auto --dist loadfile. Two runs, ~1 h, and every idea below gets a before/after. - Benches behind a marker (
@pytest.mark.bench; default run-m "not bench";python -m pytest -m benchis the slow truth). Candidates are the durations list above. Which run is “the floor everyone stands on” is Architect’s floor — PROP, with this page as the evidence pointer. Until decided, no test moves. --dist loadfileinpytest.iniso module fixtures run once. Cheap, reversible, measurable.- Second seam for the pair stores (
store_split_v1.initialize_store), same shape as PATCH-575’s: one pristine pair per process, copied. ~70 s serial. Troll’s Hole if idea 1 confirms the number. - Git chattiness on the seal/sign-out path (product code:
capability_bundles.release_writer_worktree,orphan_standing_worktree,identity_git). Count the distinct facts one seal needs against the 27 calls it makes. Each avoided call is ~25 ms × every test that seals. Troll’s Hole, but it changes production behaviour — imp with care. PRAGMA synchronous=OFFfor brand-new store builds (Alloy: 17× on the walk itself). Touches a live-store durability premise — Architect’s.- A name for the shipped seam no-op in
migrations.py(goblin concern, TRACE-2244) so tests pin the real thing. - Do not add a psutil sampler yet. It answers “busy or waiting” — only worth asking if idea 1 shows a wide floor rather than a tail. If it does: a 20-line one-shot sampling CPU% and disk bytes per second during a run, reported in a FIND, not promoted.
Start here (next Troll)
python corpus.py story FIND-1888 --full # the canary hole and its numbers
python corpus.py story PATCH-593 --full # what landed; "Seen in the Hole" carries the goblin's concerns
python corpus.py story FIND-1875 --full # Alloy's measurement and the nine reds
python -m pytest -q -n0 --durations=0 > .work/durations-serial.txt # idea 1; ~15–45 min
python -m pytest -q --durations=60 # the parallel tail, ~5 min
Then decide with Human whether idea 2 is a PROP tonight. Everything else is a Hole you can name from this page.
One-shot probe (not promoted; recoverable here)
Run from the repo root as
python -m pytest .work/troll-probe/test_canary_probe.py -q -n0 -o addopts= --basetemp=.work/pytest -s:
import sqlite3, time, json
from unittest import mock
import tools.ops_command_canary as canary
def test_probe_a_with_fixture_assert():
opened = []; original = sqlite3.connect; tripped = []
def connect(database, *a, **k):
if not ("ops-command-canary-" in str(database) or str(database) == ":memory:"):
tripped.append(str(database)); raise AssertionError(f"fixture would trip on {database}")
opened.append(str(database)); return original(database, *a, **k)
s = time.perf_counter()
with mock.patch.object(sqlite3, "connect", connect):
report = canary.run()
print("\nA WALL", round(time.perf_counter()-s,1), "opened", len(opened), "tripped", tripped[:3], "status", report["status"])
def test_probe_b_differences():
from tools.corpus_ops_v1.persistence import migrations
with mock.patch.object(migrations, "_seed_fresh_store", lambda path: None):
report = canary.run()
bad = [l for l in report["leaves"] if l["classification"] == "difference"]
print("\nB status", report["status"], "differences", len(bad))
assert False
Pointers
FIND-1875, FIND-1880, FIND-1888, PATCH-575, PATCH-593, conftest.py
(_pristine_store, _seed_fresh_store, _bind_pristine_store_seam),
pytest.ini, tools/ops_command_canary.py (run, deterministic_runtime,
_walk_fresh_store), tests/test_ops_command_canary.py (full_report),
tools/corpus_ops_v1/persistence/store_split_v1.py (initialize_store),
tools/capability_bundles.py (_run_git, _git_ok, identity_git,
release_writer_worktree), library/code-troll/2026-09-14-alloy.md.