capabilities: the non-web race machinery, part by part

  • 62solver-library modules57,767 lines
  • 2,079tests
  • 463replay fixtures
  • 9race-day handler files
  • 10tool articles

counted from the repository at every build.

this is our internal capabilities document as it stood on day 31, the last day, published with its web parts split out to their own page (web capabilities). agent ids are replaced by roles (the parent, the backup parent, a lane fixer, a helper, autopilot), and private hosts, addresses and credentials are removed; everything else is the document as we use it. the section numbers are its own: the posts and the architecture page cite them ("7.7"). each part says what it does and where it lives in our repository, as the system is now; the history of how it got there is in the day posts.

1. overview

the event releases one day of challenges at a time. each challenge has three separately scored tiers: an untimed single_solve, a timed time_trial, and a timed multi-variant mastery. a central game master (gm) assigns material, grades answers and returns flags.

the workspace lets any number of agents (the runners' agents, their subagents, or a human) work one day's challenges in parallel from a single checkout. they coordinate only through files, never by talking to each other directly:

  laptop ──https (paired bearer, keep-alive)──▶ the game master
     │        fallback only: vpn -> remote vm (ssh) -> gm
     │
  ┌──┴──────────────── one checkout, shared filesystem ────────────────────┐
  │  tools/comp/kickoff.py   09:00 on the gm's clock: plan, go signals     │
  │  tools/comp/autopilot.py auto.<lane>: no llm, yesterday's solver first │
  │  agents: parent, backup parent, <lane> fixers, helpers wN, auditor     │
  │     │  fixer: next -> fix solve.py -> autopilot re-runs, runs tiers    │
  │     ▼                                                                  │
  │  coord.py ── sessions/ (tasks, claims, assists, mail, intel, runner    │
  │     │                   states, locks, day markers, archive)           │
  │     ▼                                                                  │
  │  gm.py ───── logs/ (session.log, flags.json, tasks/flagboard logs,     │
  │     ▲              kickoff-dayNN.log, scoreboard, autopilot-dayNN.log) │
  │  ctfkit.hlib (shared solver library) ◀── challenges/dayNN/<id>/        │
  │  ctfkit.replay / dryrun / delta ◀── tests/fixtures/ (graded runs)      │
  └────────────────────────────────────────────────────────────────────────┘

runners (this document names harnesses and models by role, never by product):

design rules:

2. components

component path role
game master client gm.py transport, gm api, logging, flag recording, stop guard
coordinator coord.py task registry, claims, dispatch policy, assists, dashboard, mail
preflight tools/comp/preflight.py direct gm route, clock skew vs the gm, vpn/remote vm fallback
kickoff tools/comp/kickoff.py release watcher on the gm's clock: edge pin + clock sample at t-65s (3.6), the previous day's archive at t-60s (4.2), warm slots, poll, plan, go signals; live comes once every slot lane's single is claimed, flagged or carries autopilot's own gaps / crashed row (slot_singles: every lane but coord.OPTIONAL_LANES; autopilot_reported counts a slot that gapped and released its claim before the plan landed, so the loop never runs its full 3.0s for it); from t-5m the headless fixer launch for every open lane with no live fixer (6.4)
autopilot tools/comp/autopilot.py llm-free release pass: per lane claim, scaffold, dry-run, submit, chain tiers; follows a released single: re-runs an agent's saved fix to it from the warm process (full submit, chain on correct, else a re-ran the changed solve.py gaps row; fix-drill), and chains its tiers once anyone wins it; reacts only to the files a solver's behavior comes from (solver_files) and holds the next tier while its code is the code a run of the challenge just crashed with (chain_waits; 4.3); its release output goes to logs/comp/autopilot-dayNN.log with a date on every line; --check readiness (parallel, stamped per commit)
rehearsal tools/comp/rehearse.py offline release: newest fixtures served as day n by a fake gm behind gm._call, real autopilot -> runner chain in a temp sandbox
solver profile tools/after-action/profile_solvers.py every non-web fixture solved offline under cprofile: wall per set per lane, the hottest ctfkit functions, the same bytes parsed again (and the seconds wasted), every subprocess; after-action stage 5.1b; reports land in docs/after-action/reports/cross-day/
day simulation tools/after-action/simday.py a past day against chosen tooling (a git ref + patches in a worktree): local https fake gm with the gm model (release flip, slow first call per socket, latency, 429 window, deadlines), real kickoff + autopilot slots, scripted helpers or real llm fixers (--llm: their prompts, spawned by the parent), per-tier report vs the real day; --clean once the day's sims are done; --break <lane>[:mastery] takes the short lane name (binary matches binary-reverse-engineering-day-NN-* as a prefix, a miss is a message, not an indexerror); --headless lets the tree's kickoff launch real headless fixers (6.4; HUNTRESS_HEADLESS_FIXERS, 0 by default in a sim); a fixer's prompt is the tree's docs/comp/fixer-prompt.txt
live retest tools/after-action/testgrounds.py any exact problem, regex subset, lane or whole day against the real gm (an organizer fix, or baseline vs treatment with --ref/--as-of, patches, overlays, suites and compare-suite): one isolated git archive tree and worker per base; web runs on the second runner only; one untimed attempt per timed tier; the same gated record every run (8.4)
launchers start-<runner>.sh (one per runner), tools/comp/start_at.py race-morning parents: the primary parent on the parent model (--model <parent model>), the backup parent with its usage guard; start_at.py <runner> --at 08:41 waits and starts one with "read start.md" (6.2); --model-main-effort sets the backup parent's own reasoning effort (HUNTRESS_<RUNNER>_EFFORT; --web-fixer-effort, its old name, still works); the parent's fallback after a classifier stop is the fallback model (/model <fallback model>)
headless fixers tools/comp/fixer_launch.py, docs/comp/fixer-prompt.txt, docs/comp/helper-prompt.txt a lane fixer as its own detached headless process (the fixer model, bypassPermissions), launched by kickoff from t-5m, by coord.py spawn --launch, or by hand; the prompt files are the one copy spawn --prompts fills for the parent's agent calls (6.4)
nightly gate tools/after-action/nightly.py the end-of-day gate as one command: promote the day's correct runs, fixes pending, replay --tables, nothing uncommitted, autopilot --check + the stamp; stops at the first failing step with what to do (6.5)
gm edges tools/after-action/gm_edges.py the same read-only gm call over the direct route, the vpn interface and --proxy socks tunnels, n samples each: cloudflare edge (cf-ray), dns / connect / tls / ttfb / total medians, the gm's own Server-Timing durations (3.6)
usage guard tools/comp/<runner>_usage_guard.py watches the second runner's structured rollout events and hands off immediately on quota exhaustion
command logger tools/comp/logcmd.sh, tools/comp/web_posthook_log.py logcmd.sh tags + caps shell output into session.log and runs the solved guard first; the second runner's posttooluse bash hook logs web racer commands to logs/comp/web/session.log without relying on the racer to invoke the wrapper
session-log hook tools/comp/session-log-hook.sh same, automatically, for the primary harness; names a command by the inline HUNTRESS_AGENT=, else the name its subagent session (agent_id) set inline earlier (logs/comp/.hook-names), else the exported one; a session id never renames an exported parent (a resumed parent's map can be stale)
solved guard tools/comp/solved_guard.py, .<harness>/hooks.json pretooluse bash hook for the primary and second runners' web racers, plus a logcmd pre-check: refuses an agent's next shell command once its task was won by another agent, while the day's board is open (a day task in neither flags.json nor completions.json); the agent is resolved as the hook does (agent_of: a fixer's unprefixed command is never judged as the exported parent) and the message names it; coord.py and the read-only gm.py status/list/openapi/edge/route/wire always pass; a command naming a saved run dir (runs/<stamp>-<tier>-<attempt>) is judged on that run's challenge and tier, not the task's (a uv run replay of an open mastery run passes once its time trial is won). start_at.py passes --dangerously-bypass-hook-trust for the vetted project hook; the hook matcher is bash only, so apply_patch edits are unaffected
web timeline src/ctfkit/web_timeline.py reads logs/comp/web/events.jsonl to report provision, first capture, claim-to-solver orientation and edit-to-replay durations by tier; uses a process's monotonic stamps where available and utc stamps across processes. command logs do not carry those phase boundaries
process control src/ctfkit/proc.py run (subprocess with a process-group kill on timeout), time_cap (sigalrm soft cap + hard-kill sidecar watchdog); bounds in docs/comp/timeouts.md
file memo src/ctfkit/filememo.py a reader's parse of a rarely changing file (fixtures, intel) by path, mtime and size under a tag, kept outside ctfkit.hlib so autopilot's solver reloads keep it (the label vocabulary, known_families, 7.2)
score race graph tools/after-action/scoreboard_graph.py (+ .html template) a self-contained page of every team's scored events at the millisecond (all days, each day's release sprint, team search); team.dev is always dracula purple and always drawn. it builds its views in the browser from the raw scoreboard, so the same code serves the saved file (offline) and <output>-bookmarklet.html: a bookmark that opens the graph in a new tab of ctf.huntress.com (the scoreboard api allows no cross-site reads), where refresh reads the live board and moves to a new day once its 09:00 et release is on it. a lane / tier bar under the day tabs (any subset of lanes, any subset of single / time trial / mastery): the day's places, clears, points and every curve then count only those questions; a lane-race table (every lane of the day: our clear and lane place, the fastest clear, the gap, the teams that cleared, per-tier times against the first solver) and a question-race table (first solver, ours, gap, our place, solvers) under the finish detail; the choice is remembered in the browser; the payload block is checked in node on the cached scoreboard
cold offload tools/after-action/offload.py end of day: past days' challenge files/, harvested variants and an archived session day's session.log + scoreboard/ to the smb share behind symlinks (sha256-verified copy, then swap); an archived day's intel, mail and markers stay local (the race reads archived intel); localize brings back a day offloaded whole
fix mailbox src/ctfkit/hlib/fixes.py, src/ctfkit/fixes_cli.py solver-local fixes.replace overrides of hlib functions, published on a correct, applied to another lane only for the same error (uv run fixes)
shared library src/ctfkit/hlib/ reusable solver building blocks + timed-tier runner
cross-lane resolver src/ctfkit/hlib/crosslane.py a field one lane cannot name, answered by another lane's reader of exactly that name (7.5)
replay src/ctfkit/replay.py offline regression over saved runs / fixtures
dry run / delta src/ctfkit/dryrun.py, delta.py prior solver on today's bytes per field; inherited vs novel ids; a crash row is <type>: <message> @ file:line, an attributeerror's message followed by what the object has, one level down
race telemetry src/ctfkit/timeline.py, scoreboard.py, dashboard.py per-tier phase timeline, public scoreboard puller, live dashboards
scaffolder src/ctfkit/newchal.py creates challenges/dayNN/<id>/ from the lane's newest solver, saves the single assignment plus an exact artifact manifest; carries the prior's handlers.py (and its own carried.py) into carried.py beside it (7.8)
decoders src/ctfkit/peel.py, flaghunt.py layer peeling, encoded-flag search
solvers challenges/dayNN/<id>/solve*.py one per challenge; id as argv[1]
fixtures tests/fixtures/dayNN/<id[__tier]>/ graded past runs used as tests

3. game master client (gm.py)

3.1 transport

primary: direct https to <gm url> with a bearer token paired once through the remote vm (<token file>). each process keeps one keep-alive connection, so a process warmed before a timed start (warm()) never pays a tls handshake on the clock. the fallback is ssh to the remote vm over the player vpn, curl from there (multiplexed ssh, ~0.5s a call); the vm costs credits and stays off unless direct is down, so the fallback is built to cost at most one 3s ssh connect (SSH_CONNECT):

the release-critical list is deliberately different: list_challenges_direct ignores stale route state, uses a short request timeout, and never waits for the ssh fallback. a successful direct call clears stale degradation. kickoff also calls warm(fallback=False); an application response such as release_unavailable proves the transport is alive and does not degrade it.

3.2 api

function / cli op behavior
list_challenges(day, lane) / list released challenges; id, mode, day, lane, name
list_challenges_direct(day, timeout) release-only direct list with a bounded timeout and no remote vm fallback
status(id) / status attempt state, deadline, current artifacts[]; an unstarted untimed single (409 start_required) is started instead of failing
start(id) / start start or resume; returns questions, answer_ids, download info. starts the clock on timed tiers
submit(id, answers) / submit CORRECT (+ flag), incorrect, or partial accepted + remaining
probe(id, answers) / probe submits each field alone, giving per-field feedback against the all-or-nothing grader; a 429 resends once after the team window (3.5); a json scalar value ({"x_count": 27}, true) is its compact json text (answer_text), so variants() never calls .strip() on an int
retry(id) / retry renews an expired timed attempt; material and layout may change. after the last fresh variant it follows the gm's next_op: practice (or a 409 retry_variants_unavailable): op=practice continues untimed on the same bytes and still completes the mode; a ranked rerun never practices
reset(id, why) / reset <id> --why a fresh live target for a broken web target in untimed play (it ends a running ranked run). web only, and gated so it goes out only when needed: the roster and an unsolved challenge; a 120s cooldown and two resets a challenge a day (a third is a gm incident); no other process running the attempt; read-only status shows an attempt that is not preparing; a live ranked run needs --end-ranked; then web_live.probe_targets gets every assigned url (no redirect, two tries) and refuses when every target answers below 500, unless --state attests a wrecked live target (mailed to the web parent). the post is {op: reset, challenge} straight to the start transport (never run_for_time, never switch_target); the fresh assignment replaces the single's start.json, the reset is recorded in sessions/web-resets.json and intel. the evidence comes from the capture: web_live.target_down (no response under a transport error, or only http 5xx) turns a missing proof into web: TARGET DOWN ... with the exact command; autopilot re-captures a reset single (recapture) and re-runs a failed tier whose target was reset after the run (follow_failed)
artifact(id, delivery) / artifact legacy index-based fetch
assigned_artifacts(id, resp) fetches exactly what a response assigned (see 3.3)
validate_assignment(resp) warns when questions[].id ≠ answer_ids (stale questions after a retry)
warm(fallback=True) opens the route before a start: tcp + tls on the main and fetch sockets, no gm request (a ping is one more call in the team's per-minute 429 budget); best effort, with optional ssh fallback suppression for release startup. a connect does not warm the gm's slow first call on a socket; the kickoff's and slots' authenticated lists do that (6.2). warm reads the edge pin kickoff set at t-65s but never probes, and re-dials a live socket that does not reach the pinned ip (_off_pin: the skew sample's socket, a fetch socket dialed before the pin) (3.6)
edge_pin(force=False) / edge [--force] probes each ipv4 of the gm's name (two gets on its own socket) and pins the fastest healthy one, with the near colos, for 15 min (sessions/gm-edge.json); gm.py edge prints the table and the pin (read-only; the solved guard lets it through); the probe also tries every ip a gm socket reached in the last 24h (seen), and a far pinned socket drops the pin and dials dns (3.6)
clock_skew() gm clock minus local clock: the ntp offset (ntp_offset, ms precision), checked against date-header bounds from ~25 quick pings (~±40ms); None when direct is down, so kickoff uses local time
guess(id, field, …) / guess one guesser per field, skips rejected values, capped rejections per pivot; an answer id the challenge lacks is refused naming the closest ids (with a hint when the shell passed field value as one argument: zsh does not word-split set -- $fv), and so is a --values that parses to nothing; a non-fixer's hand probes on a lane whose fixer is live stop at one accepted field (hand_cap, 7.7)
openapi() / openapi dumps the gm's api description

errors carry the gm body (curl --fail-with-body), e.g. 409 {"error":"prerequisite_required"}.

3.3 artifact delivery

the gm describes assigned files in one of three shapes: artifacts[] (with download, sha256, filename, kind), downloads[], or a single download. assigned_artifacts picks one source by that precedence and never merges them. it downloads each file once via op=download&capability=<64 hex>, FETCH_WORKERS at a time (default 6, HUNTRESS_FETCH_WORKERS), and verifies the sha256 when one is given. the fetch pool lives for the whole process and warm() pre-connects every worker (tcp + tls, no request): connections are per thread, so a pool per call would pay a handshake per download on the clock. a 429 goes through the team-window rules (3.5): no straggler hedge while it lasts, and a window the attempt's clock cannot outlast ends the attempt (AttemptExpired) so the runner opens a fresh one. capabilities expire, so on any other network failure it refreshes them once via status. files map to sets 1..n only when the file count equals the set count; otherwise every file is set 0, a combined bundle the caller splits. a download is one stream by necessity: op=download ignores Range (200, the whole file, no Accept-Ranges) and is never compressed, even when asked, so the per-file parallel fetch is the ceiling. a capability can be fetched more than once while the attempt is live (the straggler hedge relies on it).

a live target (the gm's guidance says "your assigned artifact or target"; no lane has had one): an artifacts[] entry with no download is skipped, not fetched, and assigned_targets returns it with every top-level key named like a target (target, url, host, port, instance, endpoint ...); runner.solver_questions hands them to every question as targets (web.target_url(q)). no saved assignment has one, so its shape is unverified.

for an untimed single, newchal and autopilot persist every assignment as an indexed file plus single-manifest.json. each manifest row keeps the set number, original filename and kind, so duplicate filenames and multi-artifact singles replay exactly. the runner still understands the legacy one-file layout.

3.4 side effects

3.5 rate limiting: the team 429 window

what the gm does (observed, undocumented): inside the release minute, after a few dozen of our calls, every request answers 429 with the body rate_limited and no retry-after or rate-limit header. the window clears at the next clock minute (:00). it is per team: other teams submit and score inside our window. its trigger is unknown, so there is no fixed budget to stay under; the client spends as few calls as it can and handles the window when it comes.

how every gm call handles it (_with_backoff, under _get / _post):

def _with_backoff(call, *args):
    _hold_window()  # team window open: wait, don't send
    for delay in (0.4, 0.8, 1.6, 0.0):
        try:
            return call(*args)
        except urllib.error.URLError as e:
            if not _rate_limited(e):
                raise
            if not delay:  # ladder spent: the window is open
                _note_window()  # window_until = next :00, shared
                raise
            if window_left() > 0:
                raise  # another caller found it first
            _note_429()  # cooldown_until: pauses probes
            time.sleep(delay + (os.getpid() % 7) / 20)

3.6 the cloudflare edge (gm.edge_pin)

the gm sits behind cloudflare's anycast. from this machine it lands on a near colo (bos / ewr) most of the time and, on a fraction of connections, on an ocean-away one (dub, man, gru) that adds the better part of a second to every call on that socket: the connect and the tls handshake each cross the ocean twice. one of the name's ipv4s lands far more often than the other, and pinning an ip does not pin the colo (anycast picks it per connection), so both halves are needed. tools/after-action/gm_edges.py measures the route: the same read-only op=status / op=list over the direct route, the vpn interface and any --proxy socks tunnel, n samples each, edge from cf-ray, dns / connect / tls / ttfb / total medians, the gm's cfWorker / cfOrigin durations where a reply carries them.

gm_edges.py with --proxy through an ssh -D tunnel measures the same route from another region.

4. coordinator (coord.py)

4.1 state

file contents
sessions/tasks.json registry: {id: {base, tier, lane, day, skip, name}}
sessions/claims/<slug>.json {challenge, agent, ts, since, status: working/done, flag?}
sessions/assists/<slug>__<agent>.json {challenge, agent, helping, ts}
sessions/days/day-NN.started first-start marker
sessions/mail/*.md notes (<ts.us>__<from>__to__<to>.md, created with O_EXCL)
sessions/findings.md cross-challenge lessons, grader-confirmed, append-only
sessions/intel/<base>.jsonl per-challenge notes {ts, agent, tier, kind, text, field?, value?, evidence?}
sessions/seen/<agent>.json what each agent has already seen: {intel: {base: rows}, findings: lines}
sessions/archive/day-NN/ a closed day: its mail, findings and seen cursors (once, .closed marker), the intel of every base released before the next day (tasks.json's release day, so a web base re-listed on a later day stays live), and the session.log lines dated before the next race day (4.2)
sessions/gm-edge.json the cloudflare edge pin: {host, ip, edge, near, ttfb_ms, at, query, candidates}, fresh 15 min (3.6)
sessions/patches/work/<agent>/ an assist's working copy for coord.py patch (patch --copy): nothing autopilot or a timed runner watches lives under sessions/
sessions/edit/<base or hlib__module>.json one writer per solver / hlib module (EDIT_HOLD 120s)
sessions/code-reads/*.json shared timed-run reader leases protecting one solver and the hlib modules it imported (modules; a lease without the list covers all of hlib) (CODE_READ_HOLD 180s)
sessions/timed/<id>.json the live timed attempt's holder until its deadline
sessions/guess/<id>__<field>.json the one agent guessing a field
sessions/agents/<agent>.json heartbeats {state: waiting/idle/working/reserved/launched/retired/stopped, task, ts, model?, resume?, pid?}; sessions/agents/.seen/<agent> is touched by every coord / gm command and every logged shell command of a spawned agent (coord.touch_seen): agents_live ages a record from the later of its beat and that stamp (last_seen), 300s for a working agent (LIVE_WORK), 90s idle / waiting, 180s reserved or launched (RESERVE_S; mark_launched restarts the window at the launch). a working lane fixer past LIVE_WORK stays live to LIVE_HELD = 600s while its lane's edit lock is held or its lane's single sits in chain waits (held_quiet: a long silent compose under the lock is not a dead fixer); the watcher line then says (silent Ns, edit lock held), and (resumes a stopped agent) only after coord.py stopped. coord.py stopped <name> --reason .. [--downgrade] (a parent) marks a cut-short agent stopped: not live (GONE), so spawn reserves the name again at once, with a resume note (task, claims, edit locks, last commands) that the respawn's first next prints once, and a model one rung down MODEL_LADDER (three model sizes, largest first) after --downgrade; spawn / tick print the model per name
sessions/runners.json runner availability {<runner>: {state: up/low/out, reason, since}}
sessions/dayNN.release, .planned, .live structured kickoff generation markers {day, phase, release_id, scheduled_at, created_at}
sessions/ratelimit.json shared probe slot + 429 cooldown + window_until: the team's 429 window after a caller's 0.4/0.8/1.6s ladder ran out; every process holds to the next minute before sending
sessions/.lock advisory flock, shared with gm.py

all read-modify-write operations run under the lock. json files are written to a temp file and os.replaced, so readers never see a partial write.

4.2 day start: plan

coord.py plan [--day N] is idempotent and safe for every agent to run at session start:

  1. decides the gm route (3.1);
  2. detects the latest released day when --day is omitted;
  3. archives the days before it (archive_previous_for(day), the backstop: kickoff has already run it at t-60s and the plan finds nothing left to move);
  4. registers every challenge and tier, merging into existing progress and never resetting it;
  5. auto-skips coursestack-only lanes (welcome) and a standard challenge with no timed tiers and no lane (skip + static; coord.py board prints STATIC:, a static agent plays it through gm.py (6.6); timed tiers listed later make it a lane again);
  6. renders the dashboard.

kickoff calls plan_from(day, list) with the list it already fetched, so the release path makes no second list call.

the archive: archive_previous_for(day) is idempotent and moves only what belongs before day: an intel file whose base's release day (tasks.json, else gm.day_of) is earlier; the session.log lines dated before the day's date (_split_log: an undated line belongs to the dated line above it; a second call appends, never replaces); mail, findings and the seen cursors once per closed day (the day-NN/.closed marker, or an archive that already holds them), so mail sent since is never eaten. kickoff runs it at t-60s, before any slot posts (ARCHIVE_AT, recorded as archived in its state file so a takeover does not repeat it): an archive that ran after the first slot rows (autopilot's gaps / is yours rows) would move the day's first intel and log lines into the day before, and coord.py intel would show the fixer "(none yet)".

4.3 dispatch: next

next atomically claims one task and prints its id. the policy runs over the latest day first, then over unfinished older days:

  1. breadth. while any single_solve is unsolved, only singles are handed out, so agents spread across every challenge.
  2. early tiers. if every open single is already held, the time_trial and mastery of bases whose single is solved are released instead of idling.
  3. depth. once all singles are done, every time_trial and mastery is open.
  4. mastery prerequisite. a mastery is only handed out after its time_trial is solved (the gm returns 409 otherwise).
  5. anti-hoard. no agent gets a second tier of a base it already holds while other work is free. candidates are ranked by fewest active agents on the base, then lane, then tier.
  6. assist. when nothing is claimable, the agent is assigned to co-work a single another agent has held for at least ASSIST_AFTER = 10s. longest-held first, at most MAX_ASSISTS = 2 helpers per runner on a single. stderr says assist: <owner> has held this single …, plus <owner> is editing <targets> for every edit claim the owner holds (_edits_by), so the assist does not redo that fix. singles only: timed tiers rotate material on every start and cannot be shared. an assist reserves one field family with take <id> <family> (sessions/takes/, TAKE_S = 300s, mails the owner): the family resolves to the gap fields it names, so a take covering a gap field someone else holds is refused (exit 4), and once every gap field is taken the assist note says so.
  7. parents never own. both runners' parents get no task to own from steps 1-5, only step 6's assists on held singles (no pivot or offline assists), and solve.py refuses them before any claim: a parent holding a failed tier idles the helpers who should have it. nor an assist on a single whose lane fixer is live and not stalled (live_fixer_on; STALL_S = 90s): the parent's empty next prints fixer <name> is live on <id>; mail a value with coord.py note, a recipe with coord.py patch (parent_routes), so a parent's hand probes never win a single whose handler nobody writes (gm.hand_cap is the gm.py half, 7.7).
  8. lane fixers. <runner>.<lane> (coord.FIXER_LANES: warmups, metadata, mobile, script, binary) is bound to its lane (bound_lane): _next_for_fixer gives it its lane's single, then a timed tier of its lane to fix while autopilot owns the lane (autopilot_owns: a fresh sessions/autopilot/<lane>.json beat from a live pid): one whose attempt failed (crashed, claimed), or one whose live attempt the runner marked (gm.attempt_losing: its first incorrect submit or a set that raised, then each field lost or won by an alternative; the bytes are saved at that moment, no claim, re-handed until a solve.py is saved; follow_failed's fixed_run still guards the fix). with no live autopilot it runs the tiers itself. steps 1-6 never apply.
  9. the solver follows the flag. a single won by hand (the fixer's probe or a parent's) whose time trial nobody runs yet (coord.chain_waits: autopilot owns the lane, the tier is unflagged, no live attempt, the winner is not auto.*) waits on the lane's solve.py. coord.solver_next_step then tells the lane fixer (or the winner) CHAIN WAITS ON solve.py ... your next call: coord.py edit, then Edit: from gm.py probe / guess (exit 0, never "stop: run next"), coord.py check (exit 0), intel --value and next (item 0 of the fixer's order); the solved guard lets the fixer's commands through, so a "stop: already solved; run coord.py next" never sends the fixer away from the edit the tiers wait on (simday --llm drills the hand-off with live fixers).
  10. a fix lands inside the live attempt. a rejected timed submit is bisected and its alternatives tried while the clock allows; autopilot sets runner.REFIX for every tier it chains (autopilot.fixed_solver): between the bisect's submits, at most once a second (REFIX_POLL_S), a changed solve.py whose files are settled (unchanged across one poll) re-solves the attempt's own bytes (runner._refix_answers: every set answered, nothing unsure, at least one answer changed) and its unaccepted fields go in on the same attempt (_FixedMidAttempt ends the bisect), instead of waiting for the clock to run out.

next also prints the lane brief (docs/comp/lanes/<lane>.md, lane from the task registry) the first time an agent gets that lane in a day (cursor in sessions/seen/<agent>.json); brief <id> reprints it; an unknown lane gets the cold-start pointer instead. prep for fields no question has graded lives beside it in docs/comp/lanes/<lane>-prep.md, never printed: the brief points to it (a brief that carried its prep grew past the page cap under the clock). each brief carries its lane's ## already generic (what the registry covers) section (it lives in the brief, not agents.md, which loads into every primary-runner agent and would give each fixer four other lanes' registry history; agents.md's table keeps the routing columns). a lane fixer reads its brief, prep file and the new-family guide before release: each wait-live --next call returns one page at once (exit 5), and the release handout then carries only the task. a page stays under 8000 characters (coord.PAGE_CHARS; a longer doc goes in parts, continued: <doc> page i/n) and the project raises the agent harness's bash output cap to 64000 (.<harness>/settings.json bashOutputMaxChars): the subagent default cuts a page at ~10k.

calling next again ends the caller's current assist. wait-live --next blocks until a valid structured 09:00 marker for that day is live and then claims in the same call; stale touch files cannot release it. a parent's wait (*.main) returns within 30s of the live marker however long its --max-wait (PARENT_LIVE_WAIT_S: a stall wake must not wait behind a pre-release wait). a spawned agent's bash tool backgrounds a call at 180s whatever timeout it carries, and the task it printed is then read a turn late: coord.wait_cap cuts a longer wait-live --max-wait to 170s (BASH_TOOL_CAP_S) and says so (exit 5 = run the same command again); agents.md, start.md and the spawn prompts ask for 170, and tests/test_wait_cap.py refuses a runbook line past the cap. autopilot slots also require the release generation passed by kickoff.

autopilot on a single. workers (auto.<lane>) claim the singles before agents are released. a single a worker cannot finish is released with its gaps posted as intel, and the lane worker keeps following it:

the chain gate (autopilot.chain_waits, runner.CHAIN_GATE): each run a chain makes records the digest of the code it ran (solver-stamp.json in the run dir, 8.1: solver_digest over solver_files, content, not an mtime a copy can move). the chain asks the gate before each next tier: a tier whose challenge has a run that crashed with that very digest (a solver_error, or a raised set in a run that still submitted its other fields) waits (chain waits: solver unchanged since <run> crashed, CHAIN_WAITS keeping the code_stamp it began waiting at) and follow_failed runs it on the next settled save. a re-roll is not gated (chain_from(gated=False): the solver as it stands); web keeps its own recapture rules.

autopilot is the only gm client for timed tiers. while autopilot_owns(lane), gm.start (timed) and gm.retry raise AutopilotOwns for any other agent (HUNTRESS_TIMED_HAND=1 overrides; gm.RERUN and a live hold of one's own pass), runner.run exits 4 on a timed tier and never chains, and next hands fixers fixes, not starts. autopilot's in-memory solver lease (code_read_claim(skip_challenge), row in_memory) does not block the fixer's edit of that solve.py, so a fix lands while the attempt runs; follow_failed re-solves the saved run when it ends. tools/after-action/simday.py --break <lane>[:mastery] --think S drills it; tools/after-action/drillgate.py gates it nightly (one break per lane plus a rotating mastery break, each inside 15 + 2*think + 15s).

staffing. before 09:00 coord.py spawn --runner <runner> reserves one fixer per solver lane whose day is open and has none live, nothing else (--prompts prints each reserved name under === <name> === with its filled prompt, docs/comp/fixer-prompt.txt for a lane fixer and docs/comp/helper-prompt.txt for a helper, so the parent pastes it into the agent call and composes nothing; --launch starts tools/comp/fixer_launch.py per name instead, 6.4. reserve_fixers / mark_launched count a name reserved or launched inside RESERVE_S as live, so the parent's spawns and kickoff's headless launch never start the same lane twice); a helper only for work in no solver lane (work_available counts nothing else, so the second runner's assist slots and helper demand exist only there). a fixer is never idle supply or surplus, retire refuses it while its lane is open, and wait-live --next exits 2 once its lane is done. with the primary runner low/out, the second runner's spawn --runner <runner> reserves its own lane fixers instead. the parents route and assist but never own, and an idle parent is no supply in demand and holds no second-runner owner back. the second runner's auditor is started only after the first failed timed run.

the web lane. web is an optional solver lane: web-worker is reserved every day, any web- id prefix is its lane (never webassembly-), a day without web triggers no re-plan, and the fixer waits and retires at the board finish (_day_complete()). a first web day starts from the seed (dryrun.SEED_SOLVERS: src/ctfkit/templates/solve_web_lane.py = web.answers): autopilot claims the single, scaffolds it and runs it like any lane's solver, and a web fixer or a base-bound web worker on the fallback model fixes the gaps it posts; later web days start from the newest web solve.py (dryrun.lane_solver and coord.lane_modules match by solver lane, so a renamed web id keeps its lane). flags, the board and autopilot's name (auto.web) label it web whatever the id's prefix. the catalog watcher (tools/comp/catalog_watch.py, started at live) holds the board open (_day_complete() false) for the second runner's agents only while it runs; its first poll goes out at once, the next after the release minute (+62s), then every 20s (--interval 20). a primary-runner agent with no lane bound skips the web lane in _day_complete() and open_crashed(), so it sees the day complete once every non-web lane is done, and a lane fixer whose lane is complete retires past another lane's failed timed tier, which it never offline-assists. a primary-runner base-bound web worker on the fallback model is web's, never "a primary-runner agent off web" (_model_off_web): it counts the web board, a failed web tier holds its retire until its own base is done, and its stalls route to the backup parent. web fixers: the second runner's spawn --runner <runner> reserves three unbound names before release (WEB_WARM: a spawn-to-assignment hand-off is far slower than the chain); at release each listed web base binds one (web_base in its agent record), more bases reserve new names, and a warm fixer still unbound once the catalog window closes with every open web base bound elsewhere is surplus (web_surplus): its lane counts as done, so wait-live exits 2 and retire lets it go, and tick --runner <runner> prints it as surplus: <name>. a day with no web listing keeps the lowest warm name as the watcher for a late listing.

the watchers. the primary runner's spawn watcher is web-blind: sweeps_lane keeps web tiers out of the primary parent's stall sweep (the backup parent's tick sweeps them, only_web while the primary runner is the overall parent), web_only_wake drops any wake or mail line that names only web challenges (a cross-lane incident still wakes), and lane_parent addresses a web tier's stall / ceiling / autopilot crash / hard-kill mail to the backup parent, never escalated to the other parent (web wakes would make the primary parent read lane evidence it never acts on). the primary parent keeps coord.py spawn --runner <runner> --wait active. its "day complete" exit (2) needs a row of its lanes: a plan for today with no non-web row before the day's live marker exists is "nothing planned yet", never done, and the exit is logged to session.log with the planned day, the race day and the live marker, so the next one is diagnosable. it checks local demand every 0.5s, reserves names atomically, and excludes idle agents from a runner marked low or out so unavailable capacity cannot hide real work. every 5s it also sweeps timed stalls (no start, no assignment, no result, deadline passed, rejected, solver timeout / crash) and open gm incidents (re-mailed every 120s), and exits 6 with wake: lines when something is urgent for the parent. the backup parent has no background wake-ups: coord.py tick --runner <runner> [<id>] does the same sweep, spawn reservation, mail read and task check in one ~60ms call chained onto its commands (exit 2 / 6 / 3).

4.4 claims and recovery

4.5 sharing

three channels, separated by scope and trust:

intel (per challenge, all tiers). sessions/intel/<base>.jsonl collects notes that help another agent solve faster, before the challenge is solved. every note carries its trust level:

kind source meaning
confirmed gm.py, automatic the grader accepted this field value (single-field submit / probe), or the whole answer set (correct)
rejected gm.py, automatic the grader refused this field value: don't retry it
observed agent verified from the bytes; --evidence (how it was checked) is mandatory
idea agent untested lead, shown as unverified

agents cannot write confirmed or rejected. intel <id> observed ... --value submits first (7.6) and writes the note even when gm.py refuses the submit (rule 0, the hand cap or the guess cap: exit 4, submit_held): the note is how the value reaches the fixer. a multi-field incorrect cannot be attributed to a field, so it is not recorded. coord.py intel <id> prints the notes grouped by trust, with rejected values merged per field. next prints them, plus all current findings, when it hands out a task, so an assisting agent starts from the owner's progress.

staying current while solving. agents call check <id> between steps. besides the solved signal (exit 3), it prints every intel row and finding added by other agents since this agent's last look, then advances a per-agent cursor (sessions/seen/<agent>.json: intel rows seen per challenge, findings lines seen). next, intel <id> and findings advance it too, so nothing is shown twice; cursors reset at day rollover. a helper therefore picks up the owner's new observations and rejected probes within one step. the dashboard shows the intel count per challenge ([N]).

findings (cross-challenge). coord.py finding appends grader-confirmed lessons that apply beyond one challenge (answer-format conventions).

notes. note <agent|all> "<msg>" / inbox [--all] carry coordination messages (hand-offs, who takes what).

at end of day, confirmed and observed intel plus findings are promoted into the huntress.md playbook; the next day's kickoff (t-60s, the plan as backstop) archives all three (4.2).

patches. an assist's fix to a held solver goes as coord.py patch <id> <target> <copy>; patch --copy <id> <target> first makes the working copy under sessions/patches/work/<agent>/ and prints its path (a copy in the challenge dir would be re-solved onto the live attempt by autopilot's watch, and the race guard refuses the scratchpad).

4.6 dashboard

logs/comp/tasks.log is re-rendered atomically after every claim, done, release, heartbeat, reconcile, plan and flag. it shows:

uv run dash (the soc view, src/ctfkit/soc.py) keeps one row per tier in its challenges and mini scoreboard panels while they fit; when they do not (several web bases make two dozen tier rows), a base whose three tiers are all won folds into one all 3 row at its mastery's place (the scoreboard row shows the three ranks), live and open tiers keep their rows, and both panels fold by one shared rule (collapse_over).

the live views are src/ctfkit/dashboard.py (rich, alternate screen, dracula):

the board projections: soc._day_board and dashboard.projected_board project a local flag only under a question id the board lists (_qid_in: the _qid form, then shorter slug suffixes with and without the tier word), so a flag the board scores under another id (the organizer's one-off static challenge) never adds phantom points to the top 10.

coord.py watch / coord.py flags delegate to it when rich is importable (uv run) and otherwise fall back to an in-place ansi redraw.

4.7 flag scoreboard

coord.py flags renders logs/comp/flags.json into logs/comp/flagboard.log, oldest day first. for comparing days and agents it shows:

it runs live by default: redraws every 2s with dracula accents (time since day start cyan, took yellow, agent purple, flag green) and marks flags won in the last 2 minutes with a bold yellow >. --once prints once; --backfill asks the gm (status still returns started_at after a solve) for any entry missing timing.

5. logging

log writer format
logs/comp/session.log gm.py (every gm call), the session-log hook (every primary-harness shell command), tools/comp/logcmd.sh (the second runner, humans) [ts] [agent] …
logs/comp/flags.json gm.py on CORRECT / day start {days: {NN: {started, by, source}}, flags: [{ts, agent, challenge, base, tier, day, lane, name, flag, started_at, first_started, attempts}]}
logs/comp/flagboard.log coord.py flags (and every win) per-day scoreboard (4.7)
logs/comp/tasks.log coord.py dashboard (4.6)
logs/comp/kickoff-dayNN.log kickoff millisecond release phases and direct-list failures for that day
logs/comp/scoreboard/latest.json (+ dated copies) uv run scoreboard / dash score the public scoreboard payload
logs/comp/autopilot-dayNN.log kickoff / autopilot per-lane release decisions, one file per race day, every line dated (a day with no dated file falls back to the shared dateless autopilot.log); dashboard, timeline, agent-timeline, afteraction and tidy read it
logs/comp/fixers/<name>-<date>.log, logs/comp/fixer-launch.log tools/comp/fixer_launch.py, coord.py spawn --launch a headless fixer's own output; each launch (6.4)
docs/after-action/reports/dayNN/timeline.md uv run timeline --write per-tier phases, gaps, time sinks, rival deltas

session.log is split into sessions/archive/day-NN/ by the next day's archive (kickoff at t-60s, the plan as backstop): the lines dated before the race day move, the day's own stay live (4.2). every [gm] RESP line ends with the cloudflare colo that answered ( [edge XXX]), every new gm socket logs dial <ip> -> <colo> <ms> (3.6). for a colored tail: tspin -f logs/comp/session.log (tailspin with a dracula + huntress keyword theme in ~/.config/tailspin/theme.toml).

the session-log hook also logs one # <Tool> <path> line for a named agent's read / edit / write, so the time between an edit lock and the next command leaves a trace.

both shell loggers cap the persisted output at 64 kib. logcmd.sh streams the command's output to the terminal through a small python filter, keeps only the first 64 kib in memory and appends to the log after the command exits: tee'ing straight into the log would let a command such as rg session.log read its own output and grow the file without bound.

6. tools

6.1 tools/comp/preflight.py

run before every session. each check prints [ok] or [!!] and, on failure, the exact fix command:

an authenticated release_unavailable outside the release window counts as a healthy direct transport; skew is simply reported as unavailable. preflight does not ask for the remote vm solely because the application has no live release.

a direct read timeout is tried once more before it counts (one cold timeout is not a dead route). it exits 1 when the gm is unreachable. a probe that still fails while any agent logged a [gm] RESP in the last minute (recent_gm_resp, RESP_FRESH_S = 60, today's session logs and archives) is [ok] ... the session log has a gm RESP Ns ago: the route answers and the route line says direct (gm RESP Ns ago); clock skew, api contract and live cycle are skipped with it (a restarted parent must not read UNREACHABLE (route: none) while autopilot is holding flags and live slots on that route).

the readiness stamp line (ctfkit.readystamp, 6.5) names the board's day from the calendar (kickoff's formula; +1 once today's board is complete after 09:00, stamp_day), never the newest scaffolded day dir + 1, says when the last ready autopilot --check ran (last_check) and prints readiness stamp: covers day N on the committed tree or why the full rehearsal (~3 minutes) will run instead (readystamp.stale: no stamp, another day's, a commit under the checked paths since it, or uncommitted edits under them).

the live cycle (one untimed start, retry, download, submit on an old solved dual-pool tier) is the primary parent's: the backup parent prints live cycle: parent runs it (the primary runner's parent) and sends nothing unless the primary runner is low / out (coord.runner_state) or --live is given. it runs once per morning for the whole team: sessions/preflight/live-cycle.json holds the parent that claimed it, its pid and its verdict; another parent inside LIVE_SHARE_S (20 min) prints live cycle: running in <agent> ... or <verdict> by <agent> at <time> and skips. two parents on the same regular attempt would otherwise race each other's submits: gm.submit recovers already_solved under gm.RERUN too (the flag from op=status, no rerun time recorded) and the runner's _accepted raises _AlreadySolved, which ends the attempt's submits at once instead of bisecting them.

6.2 release automation

tools/comp/kickoff.py runs once per day (flock on sessions/kickoff-dayNN.lock; a second instance exits, and so does a 09:00 kickoff started after that day's live marker exists, keeping the real kickoff log). the primary parent owns it; the backup parent's --backup instance tries the lock every 50ms from t-2min and, if the owner dies, adopts its progress from sessions/kickoff-dayNN.json (release id, clock, slots) so the owner's warm slots still fire. it sends no gm call before a takeover. --backup detaches itself (its own session, output appended to logs/comp/kickoff-console.log, prints backup kickoff detached: pid N) and writes sessions/kickoff-dayNN-backup.json; at t-5m the owner checks that pid is alive and, if not, logs a warning and mails both parents URGENT no backup kickoff with the command (a backup started in a foreground shell dies with that shell, unnoticed otherwise). it clears old markers, chooses one release generation, pins the gm's cloudflare edge and measures the gm clock at t-65s (08:58: its calls stay out of the 08:59 warm-up minute; pin_edge then the skew sample, 3.6), archives the days before at t-60s (4.2), performs direct-only warmup at t-20s, and launches the pre-warmed autopilot slots with that generation. every list socket (2 hedge + 2 fresh) sends an authenticated list at t-20s and t-3s, which warms the gm's slow first call on that socket. from t-0.3s the 2 hedge sockets take turns sending the direct list every 80ms, and at t+0.05s / +0.15s the 2 fresh ones send one each (sends in flight together slow each other); 6s read timeout, no socket closed for a retry, until the first non-empty answer or the give-up deadline. it plans from the successful response and atomically writes matching planned/live markers (live once every slot single is claimed, flagged or carries autopilot's own gaps / crashed row, wait_for_claims, at most 3s), waits until every send still out has answered (each logs its time), and re-plans once at +62s, after the release minute, only when a lane is missing from the list. every phase is persisted in logs/comp/kickoff-dayNN.log for the dashboard and timeline.

each lane slot lists its own lane from t+0.01s and runs that single (autopilot.lane_single). a lane whose single's id is known before any list (autopilot.PREDICTED: warmups-day-NN) also sends that start at t+0.01s and, if the gm refused it as too early, once more at +0.3s (gm.start_predicted: direct only, no failover, no resend; a refusal is none). the first answer wins: a started guess runs at once with its assignment (one(cid, started)); a list naming the guessed id while its start is in flight waits for it (2.5s) instead of starting twice; a list naming another id runs that id, started the usual way. the guess thread warms its own socket (lists at t-20s and t-4s, as the slot does): gm keeps one connection per thread, and a cold one pays the slow first call. the predicted lane's list runs in its own lane_lister thread on a socket of its own, warmed like the guess's, so a started guess runs the moment it is in hand and the list only confirms it. every slot also reads the timed tiers' label vocabulary before the release (runner.prime_labels, 7.2).

the day's holder also launches the fixers (6.4): a fixer-watch thread starts headless fixers for every open non-web lane with no live fixer at t-5m, then every 30s after the release until every non-web lane is flagged (bounded rounds, 12h at most), so kickoff stays up through the race.

the backup kickoff (the backup parent's, alive from ~08:42 but armed only at t-2m) runs a sentinel thread as well: at t-5m a primary parent with no [parent] $ line in logs/comp/session.log since the launch and no owner kickoff of its own gets the human a desktop notice and a spoken line (notify_human: osascript + say), a note to the backup parent, and the headless fixers launched at once (launch_fixers; coord.reserve_fixers never doubles a lane). a stale ready stamp is handled by ready_check from the first kickoff's start, owner or backup: autopilot --check --day N runs detached (logs/comp/ready-check-dayNN.log, pid marker sessions/ready-check-dayNN.json, one run), preflight's readiness stamp line points at that log instead of a second run, and the sentinel stamps the verdict at t-5m (NOT READY mailed to both parents and shown to the human). HUNTRESS_HEADLESS_FIXERS=0 (a sim, the tests) switches the sentinel and the detached check off together.

the parents start from launchers: the primary runner's start-<runner>.sh runs its parent on the parent model (model --model <parent model>; start.md has a main on any other model ask once for /model <parent model>), and every primary-runner agent it spawns, fixer or helper, gets the agent tool's model: "<fixer model>" (the configured name of the fixer model): a subagent inherits its parent's model otherwise. coord.py spawn --runner <runner> repeats it on stderr (stdout stays one name per line). tools/comp/start_at.py <runner> waits for --at (08:41 by default: the fixers are spawned well before kickoff's t-5m headless round) and starts the launcher with "read start.md".

a parent's first turn can be stopped by a classifier before any tool call, and with it everything the parent owns but kickoff. the parent's fallback after such a stop is the fallback model (/model <fallback model>; a normal day still starts on the parent model); start.md's late start makes coord.py spawn --runner <runner> --prompts the first command and the agent spawns the same message, then the watcher and wait-live --max-wait 30, preflight and mail after; the fixer prompt is a file (docs/comp/fixer-prompt.txt, {name} filled by spawn --prompts, parents.md points at it) that the parent pastes and never composes; and kickoff's headless launch (6.4) covers a parent that never gets that far.

the second runner's start-<runner>.sh launches tools/comp/<runner>_usage_guard.py beside it. the guard only reacts to the structured usage_limit_exceeded event, then runs coord.py runner model out; ordinary process exits do not trigger a handoff.

6.3 ctfkit cli (uv run …)

command purpose
newchal --gm-id <id> creates challenges/dayNN/<base>/, saves start.json, single-manifest.json and indexed artifacts, and copies the newest lane solver plus its unpromoted handlers as carried.py (7.8); does not start a timed tier
newchal "<name>" [artifacts] -k basic/remote/web generic scaffold, copies and extracts artifacts (pw infected), runs triage
replay [runs…] [--solver f] [--promote] [--promote-single d] [--benchmark] offline regression (section 8)
delta <id> inherited vs novel answer ids from earlier days' fixtures, then the dry run; fetches the single if nothing is saved
dryrun <id> [--solver f] the lane's newest prior solver on the single's bytes: ok / bad shape / crash per field (answers pass through runner.conform; tokens not joined by X flags a value conform cannot fix; NOTE xN re-joined ... lists values conform did fix, so a producer emitting the wrong separator stays visible; 15s cap per answer set; a one-field set whose solve_set raised takes that crash as the field's verdict instead of making the same call again; FIX AVAILABLE <fix> under a crash another lane already fixed; UNSURE <field> for syntax read two ways)
fixes list / apply <fix> <id> / pending the fix mailbox: published fixes; apply one into a challenge's solve.py (refused without a matching crash or when the solver already replaces that function; reverted unless the crash is gone and nothing new broke); solver-local fixes still to move into hlib
labels <challenge> --field … --feature … graded value vs artifact feature tables
triage <file> json report per artifact, --tokens for label guessing
timeline [--day N] [--write] per-tier phases since release, ranked gaps, time sinks, ## fixer tasks (every non-web tier an llm agent was handed, target none) (refused starts only from the runner's [coord] timed start refused record, never command text or gm payloads), rival deltas; a tier claimed before its base's flag counts 0s handoff, not negative
scoreboard [--day N] [--csv f] [--cached] public standings, margin to #50, per-tier solve times and ranks, top-50 export
dash [soc/commands/today/race/flags/score] [--day N] / dash web [--once] [--interval N] live dashboards (4.6); the web view shows lane status, claims, attempts, target requests, cooldowns, and advisor handoffs
web-inspect <saved-run/capture> [--handoff] offline map of saved web responses, forms, links, route clues, and answer-to-target associations; --handoff queues bounded analysis of the saved packet
agent-timeline [--day N] [--tail S] [--write] per-agent race timeline and runbook checks from the same records (docs/after-action/reports/dayNN/agent-timeline.md)
afteraction [--day N] the after-action evidence pack (docs/after-action/reports/dayNN/evidence.md): timeline, agents, tiers, incidents, code changed since the previous review, its open items, raw sources and the narratives to check (after_action.md). its agent checks include value in hand, not submitted: for every tier slower than 15s, each graded value of 10+ characters (any written form, json escapes ignored) from the first session.log line any agent could read it in (a command's output, an intel post, a note, a submit; a timed tier's own lines only) to the first submit of its graded form, flagged past 15s
graded add/list the grader-confirmed label tables (hlib/graded/*.jsonl): an atomic append with --evidence <challenge>:<answer id>, no lock
variants harvest/backtest/prompts/export/import after hours: pull, grade and learn from more variants of solved tiers; web harvest also admits verified external completions, then checks gm status before spending a pool entry
challenges [--list] [--day N] [--lane X] [--tier T] [--slower-than S] after hours only: replays solved timed tiers for the challenge-time boards (docs/after-action/after-hours.md)
peel <file> / peel -s <str> breadth-first decoding through encode/compress layers
flaghunt [paths] flag search across ascii, utf-16, rot13, reversed, base64/32, hex

6.4 headless fixers (tools/comp/fixer_launch.py)

a lane gets its fixer with no interactive session alive: the fixer spawn is not only an agent call inside the parent's session, so a parent that stops before its first tool call leaves no lane without one.

6.5 the nightly gate (tools/after-action/nightly.py)

python3 tools/after-action/nightly.py [--day N] [--dry-run] [--skip-check] runs every step the ready stamp needs, in order, and stops at the first failure with one line of what to do (steps done by hand miss a promotion or a table row; the gate does not):

  1. every correct non-web run of the day whose tier has no fixture is promoted (replay --promote, the earliest run per tier; --promote-single for a single won with no saved correct run: a probe, a sweep); a base that owns no <base>/<tier> task in sessions/tasks.json is no lane base and is never promoted (lane_bases: a static challenge whose id carries a lane prefix would otherwise land as a fixture the lane solver cannot read and fail every hlib --release replay and the ready check); with no task store every base counts;
  2. uv run fixes pending is empty;
  3. uv run replay --tables over the day's fixtures: every one passes and no table row was supplied by a search (8.3);
  4. nothing uncommitted under the checked paths (readystamp.edited(), "commit first");
  5. autopilot.py --check says ready and the stamp read back covers this commit.

it is the first "always" item in after_action.md and the primary runner's step, re-run after the evening's last commit by anyone (any tracked edit under the checked paths before 08:49 invalidates the stamp).

the ready stamp (ctfkit.readystamp): sessions/ready.json holds the day, the head, the checked tree, whether the check was full and when it ran. the tree key is git rev-parse HEAD:<p> over readystamp.CHECKED_PATHS (src, tools, tests, challenges, gm.py, coord.py, the runbooks and the web lane's docs, pyproject.toml, uv.lock; the full tuple in 13.7), so a commit outside them (a report, a non-web lane brief) keeps the stamp. the day it covers is the calendar's (stamp_day, 6.1). it is written only on a clean checked tree: "clean" is readystamp.edited() empty, every tracked edit under the checked paths and an untracked file only when the rehearsal would load it (a solve.py / handlers.py under challenges/, a directory holding one, anything under the code paths); any other untracked file never withholds it.

what autopilot.py --check runs: independent processes all at once, so the check takes its slowest step, not their sum: every lane's newest solver over all its fixtures (replay --tables), the script lane's generic_check.py on a full check, the entry-script check, the web and preflight regression tests, and the offline release rehearsal (tools/comp/rehearse.py). each step has CHECK_STEP_S (300s); the rehearse step has its own REHEARSE_STEP_S (480s): the rehearsal is ~245s of cpu alone and shares the cores with the lane replays.

the entry-script check (tools/comp/entry_check.py, a step of autopilot --check; tests/test_entry_check.py): the race scripts run under the shell's bare python3, not the venv, and the rest of the gate runs under the venv, so it checks three things in about a second each: the venv's editable install points at this tree's src (editable_src: a uv run in a sibling worktree re-points it); every ctfkit import in a race script (an ast walk over tools/comp/*.py and the entry points, a lazy import inside a function too) has the script's own sys.path line naming src (SRC_LINE); and each entry point start.md runs with a bare python3 (ENTRY: coord.py, gm.py, kickoff, preflight, autopilot, fixer_launch, install, solved_guard) that does not re-exec into the venv imports under python3 -S (no site-packages, as on a race shell), its main guarded so nothing runs. a script that re-execs (os.execv) is never smoked: an import of autopilot.py claims the board, one of preflight.py runs a live cycle; the venv has ctfkit, and the path rule still covers it.

the upcoming audit: the upcoming lines of autopilot --check are prep hints for the end of day (13.1 stage 4, 13.4), never a block on ready. upcoming_labels over the fixtures lists each lane's enum values a question offered that no graded answer ever used. upcoming_named(challenges/) reads every non-web day's start.json and prints upcoming <lane> record magic X: named in a question, no reader (a XXXX record magic, NAMED_MAGIC, that no hlib source and no solver of the day names), upcoming <lane> field family f: named in an answer id, no reader (a metadata property family no line of forensics / records / containers names; a binary dNN_<section>_<name> neither compound._named, after a role of up to compound.ROLE_WORDS, nor recipe.parse reads), and on each lane's newest day only (the family a timed tier or tomorrow extends; older days' members sit on the cold share) member kind .x (a suffix the questions name whose members sniff as binary, UNKNOWN_KIND) and the prompt's listed kinds (PROMPT_LIST, named in the prompt). a phrase is known when all its words share one source line (_Vocab: a registry key, a regex, a reader's comment citing the words). tests/test_autopilot_upcoming.py.

6.6 the static challenge kit (tools/static/, challenges/static/)

the organizer's "static standard challenge in the reverse engineering category" (a regular file-download ctf challenge) is scored on the coursestack page and never appears on the game master, so none of the race machinery sees it. the kit is for one static agent the human opens by hand with "read challenges/static/readme.md":

7. shared solver library (ctfkit.hlib)

distilled from the day-1/2 solvers and checked against their saved artifacts. covered by ruff and mypy --strict. solvers import it with:

ROOT = Path(__file__).resolve().parents[3]
sys.path[:0] = [str(ROOT), str(ROOT / "src")]
from ctfkit.hlib import archive, decode, elf, forensics, ioc, pe, runner, script

7.1 the solver contract

a solver supplies one function and hands it to the runner:

def solve_set(
    files: dict[str, bytes], fields: list[str], questions: list[dict]
) -> dict[str, str]: ...


runner.run(challenge_id, solve_set)

7.2 runner.run(id, solve_set)

run is one tier (run_one) plus chaining: on CORRECT it marks the task done, frees its own edit claim, claims the next tier and runs it with the same solver (single -> time_trial -> mastery; HUNTRESS_NO_CHAIN=1 opts out). autopilot sets CHAIN_GATE: the chain asks it before each next tier and, given a reason, prints <tier>: chain waits: <why> and releases the tier instead of starting it (none for an llm-run solve.py: never held). run_one, in order:

  1. lease and preflight (timed tiers): atomically acquire a shared code-read lease, then preflight every start (no result cache): replay the solver on the newest correct run or the single's confirmed fields (PREFLIGHT_CAP_S = 15s), refuse to start on any disagreement, and verify the code hash (solver dir + the hlib modules it imported) did not change. a ranked candidate equal to the confirmed value agrees and is moved first; the position it held leads the timed submit (7.6). an autopilot chain on its loaded module leases only the hlib modules and skips the challenge's own edit gate. the label vocabulary is read here, in a thread beside the start's round trip, never in prepare on the clock (prime_labels; autopilot's slot reads it before the release).
  2. start: an untimed single whose saved start (files/start.json + manifest bytes, written by autopilot or newchal) is the team's latest recorded attempt (sessions/attempts.json started_at) and matches the start's sha256 is reused with no gm call (saved_single; the solved guard still applies); otherwise gm.warm() (connect only), then gm.start (resumes a running attempt; a 429 holds for the minute, 3.5).
  3. fetch: a re-solve of a live timed attempt (a fixer's save while it is open) reads the bytes from an earlier run dir of the same attempt when every artifact's sha256 matches the reply's, never on a path match alone (saved_attempt: a resume hands out new capabilities, never new bytes); else gm.assigned_artifacts: parallel, sha256-checked fetch; a rate-limited fetch with clock left fetches the same attempt again, one the window would outlast goes to gm.retry (3.5). under gm.streaming(on_set) (the runner's StreamedSolve) a per-set bundle (binary and mobile mastery; script, warmups and metadata come as one bundle, not streamed) is unpacked in its download's worker thread (prepare: the aes bundles' pbkdf2 in parallel, so fetch in phases carries that unpack) and solved the moment it lands while other sets are still on the wire (the straggler hedge skips a download that landed during that solve). streamed outputs are merged only for the sets whose very bytes the fetch returns (a refreshed re-fetch streams nothing); a set that raised is solved again by step 5, which records the failure; a cap hit is raised at once (one CLOCK_CAP_S = 15s / clock cap a run); phases carves the overlapped time out of fetch as solve_streamed.
  4. split (split_sets): per-set downloads map directly; a combined bundle is split by explicit member markers (set2, slot-2, evidence-02.zip, blob_2.txt). each nested zip is read once (archive.extract_with_members: its flattened files and its top-level members from one read, so an aes member pays its pbkdf2 once) and the nested zips expand in a pool like per-set downloads. each set is solved under one setcache.scope().
  5. answers (answers_for on the sets step 3 did not stream, then check_answers on the merged answers: exactly the assigned ids, none empty). each answer goes through conform: an ordered_tokens token list is re-joined with the separator its answer_format names; a label where the kind is integer becomes its number; a record's one-element sequence ([184], the bare disk-partition-start-lba of a one-partition disk) asked as an integer kind answers the bare integer (_as_integer; a json kind keeps the array); 0x prefix, case and compact json follow answerfmt's reading of the syntax; every answer carries its other written forms (gm.variants) as shapes (7.6). a solver crash saves the run (assignment, downloads, error) before raising. crash-free partial submit (_crash_free): a single-set timed attempt whose set raised is halved until each crashing field stands alone (ISOLATE_CALLS_PER_LEVEL = 4 calls a halving level, plus 8, inside ISOLATE_SHARE = 25% of the clock left; past the budget the rest is marked NOT_ISOLATED) and its clean fields are submitted; the crashers stay unsent as gaps, named in the gaps row (_isolated_line), for a fix to send on the same attempt (REFIX). a multi-set attempt keeps its per-set fail-soft (a raised set of several is not split).
  6. submit: one atomic submit, or with alternatives: first one optimistic full submit when the clock covers it plus a bisect (a wrong full submit leaves the attempt open), then a grouped submit where risky fields go first and a rejected group is bisected and retried with alternatives inside the same attempt. on timed tiers the label fields of a family no earlier fixture has seen get automatic alternatives: every value the grader ever accepted for that suffix, and every t1059 sub-technique (T1059_ALL) for a guessed interpreter -attack (auto_alternatives). the vocabulary (label_vocabulary: every fixture and confirmed intel row, live and archived) is read per file through ctfkit.filememo, again only for a file that changed and across solver reloads.
  7. save: save_run writes files/runs/<stamp>-<tier>-<attempt>/.
  8. expiry and gaps: on expired: gm.retry (practice once no fresh variant is left), re-fetch and re-solve, up to retries=2; release the shared code-read lease after the last submit or error. lost_line posts a field rejected on every candidate as rejected on every candidate; bytes: <member> in <run>/artifacts (N forms: ...), the set's own bytes inside the 300 chars intel shows; uv run replay <run> of a run with a result prints what autopilot.fixed_run checks (replay.partial_report): accepted values that still come out, each lost field's tried forms against its new value, the unsure answers the gate waits on.

conventions. a regraded family's (REGRADED) solver answers [new rule, old rule]; apply_convention answers every disagreeing field by the day's rule (gm.CONVENTIONS per day, else a graded value in intel: a probe's row or a correct's answers), else by the new rule (0); the other value stays the next candidate. never a probe: autopilot's single sends a missed full submit once more under the old rule and its correct names the rule (save_rule). the solver's raw order (raw_forms[0]) names the rules, so a graded lead already in front is never swapped back. rehearsal convention-drill: the same day under either rule flags every tier with no incorrect timed submit (1 incorrect full submit on the single at most).

7.3 modules

one row per module: what it reads and the subsection that describes it.

module reads section
runner one tier (run_one) plus the chain: lease, preflight, start, fetch, split, answers and shapes, submit, bisect, save; regraded conventions; the graded lead 7.2
unsure Unsure(candidates, note): a low-confidence answer in its own bisect group (re-exported as runner.Unsure) 7.2
crosslane resolve(field, bytes, name, skip=lane): a field one lane cannot name, read by another lane's reader of exactly that name; READERS, closest, FAMILY_FORMATS 7.5
locate answers with no reader: a token stored in the bytes, a json member's value, assembled pieces; rules learned from a graded single (words, position, role, codec, header integer) 7.7
roles role routes: a field word naming one of two sibling builds, answered as the field without the word 7.7
fields the race-fix block: handlers at FIELDS = Fields(), handlers.py, carried.py, the crashed-handler fallback 7.8
inventory every reader's outputs as readings: sweep, learn, answer, the hand-off row's material 7.9
recipe the question-recipe interpreter: a selector and an op read from the question's own sentence 7.11
ids parse("set2.slot3_marker") -> set, slot, field; by_set(ids) 7.12
archive zip / 7z / winzip-aes extraction, members, named, resolve_member, set splitting 7.12
answerfmt the answer-format syntax read into a Spec (prefix, case, separator, width, unsure) 7.12
expand bounded decompression for every reader: soft checkpoints, a hard ceiling, TooBig 7.12
fixes solver-local overrides of hlib functions and the cross-lane fix mailbox 7.12
tables rows a search supplied instead of a table; replay --tables fails on any 7.12
setcache a cache for one set's solve, by bytes identity; nothing across sets 7.12
decode the warmups line decoder: day table, codecs, chain search, peel and beam 7.13
forensics the metadata property router: property_value over exact readers, header structs, family records, relations and exiftool 7.14
records one record per artifact family (ole2, evtx, hive, pcap, sqlite, disk, the zip kinds, mft, pdf, png, fonts, icons ...) 7.14
containers header records for crash dumps, hiberfils, volume boot records, ext superblocks, qcow2, vmdk and ewf images 7.14
headers raw <struct>.<field> readers per format and the word guess over them 7.14
composites the derived keys a flat record implies (dimensions, aspect ratio, seconds) 7.14
msi windows installer databases: the property table, the summary stream, every msi- property 7.14
synthesis a cross-member field the question spells out (digests, sizes, order) 7.14
media exiftool, ffprobe and ffmpeg wrappers, qr, pixel and perceptual keys, crops, sha256-keyed memos 7.14
netcap tshark wrappers: a capture's hosts and names, keystrokes, icmp payloads, http objects 7.14
winart lnk, prefetch (win10 xpress-huffman unpacked), thumbcache, pyc, wim, memory images 7.14
office ooxml macros and document info, repair_zip 7.14
mail eml and outlook .msg parsing, urls and attachments 7.14
pdf pdf objects, risky keys, pdf.record 7.14
hive registry hive walk, persistence keys, userassist 7.14
winlogs evtx events: counts, script blocks, processes, services, logons 7.14
apk the mobile puzzle json: codec chains and ciphers, split / packet / table codecs, the layer peel, the answer scan, bundles 7.15
dex a dex or apk's own classes, the dalvik interpreter Vm, its framework layer and the clean-phone stand-ins 7.15
android the binary manifest, network security configs, signer certificates and app fields by name (app_values); the jni emulator 7.15
specimen the any-language descriptor finder, the stage graph and its comparisons, ranked candidates, composite fields 7.16
script powershell and jscript answers; the role and attack tables 7.16
vbscript the codec_ / material_ decoder in any language, the control readers 7.16
batch fragment reconstruction, the expansion and control labels 7.16
python the generic descriptor plus the surface tuple 7.16
graded append-only label tables (roles, attacks, interpreters) and the uv run graded add contract 7.16
wmi filters, consumers and bindings from mof, powershell, wmic and batch echo builders; which subscription survives a reboot 7.16
carriers compiled script carriers to text (applescript, luac, autoit, else printable strings) 7.16
outcome the dynamic trace's outcome types (Outcome, Plan, Unsupported), a leaf emulate and trace share 7.17
placement where a file offset sits in an elf / pe / mach-o: its section (Region) and virtual address 7.17
compound the binary lane: solve for dNN_ fields, the hcfg carriers and overlays, the producer registries 7.17
elf elf headers, symbols, c++ info, hcfg records under any magic, disassembly, the metadata record 7.17
pe pe info (machine, bits, subsystem, imports, exports, imphash, pdb) and the metadata record 7.17
dotnet the in-house ecma-335 metadata parser: tables, heaps, named columns, config, p/invoke rows 7.17
macho mach-o thin and fat: architectures, sections, symbols, disassembly, record 7.17
wasm a webassembly decoder: sections, imports, exports, memories, tables, segments, opcodes 7.17
java jars and class files: manifest, constant pools, a full class-file parser, kotlin metadata 7.17
native go buildinfo and pclntab, rust demangling and crates, autoit, source-language detection 7.17
swift swift5 reflection metadata on elf, mach-o and pe 7.17
objc objective-c runtime tables of a gnustep-2.0 abi elf, read by section 7.17
trace the binaries' dynamic trace line: labels from the bytes, outcomes per host model 7.17
emulate a small jvm / cil interpreter building (branch, count) per host model 7.17
cil .net method bodies (il) over dotnet.tables 7.17
versioninfo pe rt_version: stringfileinfo keys, the fixed block, the translation 7.17
ioc indicators from text or bytes: urls, ips, domains, hashes, paths, wallets 7.17
web, web_live the web lane's offline artifact answers, live capture pipeline, recipe families and transports; described in web capabilities web capabilities

7.4 reference solvers

the newest challenges/dayNN/<lane>-day-NN*/solve.py of each lane is cumulative: it solves every earlier day of its lane (autopilot.py --check proves it against all of the lane's fixtures). newchal and the autopilot start each new day from it, with any handler the end of day did not promote carried beside it as carried.py (7.8). a lane with no graded day (web) starts from its seed; until web has fixtures, rehearse web-drill serves a synthetic access-log challenge (all three tiers, the gm's three artifact shapes) through autopilot: the seed alone must flag every tier, and a field it cannot read must crash, be fixed by the brief's fix-kit handler and chain from autopilot's re-run.

7.5 cross-lane field resolution (hlib.crosslane)

problem. each lane has its own field vocabulary and readers, and a challenge can ask one lane about an artifact another lane parses every day (a metadata single asking dotnet-mvid-guid about a managed dll: the metadata readers have no such property, while hlib.dotnet, the binary lane's .net reader, parses the #GUID heap).

rule. a field a lane's own readers cannot answer is offered to the other lanes' readers under exactly its name. there is no fuzzy matching: a reader of a similar name is listed in the error, never used. an empty value is no answer, and two lanes that answer differently raise instead of picking one.

algorithm (resolve(field, data, name, skip)):

  1. split the field into words on -, _, . and whitespace, lowercased. a leading family word (pe, exe, dll, dotnet, elf, so, macho / mach-o, jar, apk, android, dex) is split off: it names the formats the field is about (FAMILY_FORMATS).
  2. ask every reader in READERS except the calling lane's (skip) over the bytes. a reader that cannot parse the bytes (KeyError, ValueError, IndexError, OSError, BadZipFile) is skipped:
lane reader names it answers
metadata forensics.exact_value every exact property (PROPERTIES, STRUCTS, png / pdf tables), with the words joined by -; never a header or exiftool guess
binary compound.PRODUCERS + pattern producers the words after the family word, or all of them, joined by _; only on an elf / pe / mach-o, or a jar when the family is jar
dotnet dotnet.named <table>-<column> for the one-row Module / Assembly tables (ecma-335 names), assembly-version, assembly-<attribute> and the bare attribute name (target-framework, file-version, informational-version, configuration, company, product, title), assembly-references, metadata-version, runtime-version; mvid aliases module-mvid, and a trailing guid is allowed only on a guid column; managed pe only
mobile android.app_values every app field android.FIELDS names (manifest, network security config); apk only, family apk / android or none
  1. no reader answered and the field has a family word: if the bytes are a zip or apk with exactly one member of that family's format, ask again over that member (dotnet-... on an apk holding one dll).
  2. one distinct value: print crosslane: '<field>' read by the <lane> lane (<reader>, <file>) (the dry run shows it) and return it. several values: KeyError("crosslane: lanes disagree ...") listing each. none: KeyError("crosslane: no lane reads '<field>' by name; closest cross-lane readers [...]"), the five reader names sharing the most words.

where it is called. each lane tries its own readers first:

caller when
forensics.property_value after exact_value, before the header / exiftool guesses (through the forensics.CROSS_LANE hook)
compound.generic a field no producer, trace or registered analyzer answered
android.answers an app field app_values does not have
the script solver (_cross_lane, inherited by each day's solve.py) a field the specimen descriptor does not answer: the member's stem is stripped from the field and the member's own bytes are read
the metadata solver imports crosslane only to install the forensics hook

forensics cannot import crosslane (crosslane imports compound, which imports elf, which imports forensics), so crosslane sets forensics.CROSS_LANE = _for_metadata when it is imported; compound and android import it at module level, so any solver importing either has it installed. hlib functions never import inside a function body (a test enforces it).

adding a reader. write def _x(ws, data, name) -> Hit | None (none when the bytes are not its format or the name is not its own) and add ("<lane>", _x) to READERS. add the names to closest if they are not already in a registry it lists. a new family word goes in FAMILY_FORMATS with the forensics.file_format values it covers. tests: tests/test_crosslane.py (each lane reading another's field over a real managed dll fixture, the closest-reader crash, an empty column, the guid kind word, a disagreement, the .net identity names).

cost. crosslane links every lane's readers, so a lane's timed run leases most of hlib and an hlib edit to any reader module holds every lane's timed start until it is released: hlib is not edited during the race (1, design rules). a field read in a solve.py during the race is promoted into its lane's reader (a forensics property, a compound producer, an android field, a dotnet column) at end of day; from then on crosslane offers it to every lane.

7.6 answer shapes (gm.variants, runner.shape_forms)

the grader compares text. an identifier with several written forms is graded in one of them, and the answer syntax does not say which: a dotnet-mvid-guid and an msi-product-code-guid share one syntax ("canonical guid text in 8-4-4-4-12 hexadecimal form, including hyphens"), yet one is graded bare lower and the other only braced upper, {DD1A80D5-...}. two rules cover it:

the graded lead (_lead_first). preflight and autopilot's reproduce check pass from a solver that returned any variant, and the same slot (the same transform) leads the field kind on the timed tiers' fresh bytes: a probe-won {UPPER} single chains at once from a solver that returns bare lower, pe_cli graded pe-cli leads dotnet_clr as dotnet-clr. each learned position keeps the solver's own candidate count beside it (<key> own, OWN, _learn_lead; shown_lead prints the positions alone). on fresh bytes with another count (_carried) only the first candidate and its slots keep their rule, since an order-keeping dedup never moves them; any other position falls back to the solver's order with one lead <key>: graded as candidate i of n, m here line. so a handler's list keeps one entry per rule, duplicates too, never list(dict.fromkeys(...)) (agents.md "fixing in seconds" 7).

shapes never regroup a submit. the shapes travel apart from the solver's own candidates (answers_for(shapes=)), so they never make a field "risky": a right run is still one full submit. only a rejected group is bisected to the field and its alternatives, then its shapes, are tried while the clock allows (_submit(shapes=)); an Unsure field keeps its own candidates only. a field rejected on every form lists them in its gap note ("the value is wrong, not its shape"), so nobody hunts a new value while a shape is untried or resends a tried one. conform still rewrites only what the syntax says without doubt (hex-shaped values); shapes never reorder the solver's first candidate unless the grader confirmed another. word order is not a shape: a 2-3 word value's other orders (hcfg_section_raw -> raw_hcfg_section) go out only with gm.py guess --reorder, a last resort on a value read from the source (agents.md "when it fails" 7).

the hand path uses the same list: gm.py probe follows a rejected value with its other forms in the same call (--exact: one form); gm.py guess --values sends each value as given then its forms, never waits on another guesser's claim (a held value, not a token list) and counts a value's forms once against the guess cap (variant_key, GUESS_CAP = 25 rejected values a field); both re-read the rejected rows before every send, so two agents with the same value never send the same form twice. posting is submitting: coord.py intel <id> observed "<note>" --value "<as stored>" [--field <id>] runs that guess (the one answer id of a one-field challenge needs no --field) before the note is written; intel without --value on an open single prints the reminder, and next's assist note says a value goes to the grader whoever took the field.

autopilot's own single. its dry run keeps each field's other values (Report.candidates: the solver's ranked candidates, then the answer's other written forms); after a rejected full submit and the per-group bank, autopilot releases the claim (the owner starts at once) and sends a rejected field's other values like an assist's probe (gm.try_values), before following the single into its tiers.

tests: tests/test_shape_forms.py (guids, labels, hex case, wrappers, slashes; a right run is one submit, a wrong shape is found), the msi mutation in tests/test_mutations.py. agents: agents.md "a value in hand" (the next command submits a value as stored with every variant; a syntax reading is an idea, never observed).

7.7 located answers (hlib.locate, autopilot)

a new family's single can have no reader at all while its answer sits in the bytes: a token findable by its shape and the words before it (...ProductCode{DD1A80D5-...} in an msi string pool), a value a json member spells, pieces that assemble to the asked shape, an enum of two or three values. locate finds them without a parser; autopilot sends them like an assist's probe and learns a rule from whatever the grader accepts.

the sweep (autopilot.sweep, after the claim is released). a field the lane's reader crashed on gets locate.derived (derived: the sources below, most specific first, deduplicated, capped at SWEEP_CAP = 24 values, under gm.GUESS_CAP); gm.kind_filter drops what the field's kind cannot be and gm.try_values sends the rest, rejected ones skipped:

  1. spelled_json: the json member the question names by path (crash/trace.json), its keys the question's words name, an ordinal (first, last) or count of a list, a scalar as is (stageThree).
  2. the stored tokens (sweep): locate.token_pattern reads the asked shape from the question (guid / uuid / 8-4-4-4-12, or a pattern syntax naming an n-character hex width, with a literal XXX- followed by prefix); scan finds every match (each file, zip members one level down, ole2 streams; latin-1 and utf-16le); ranked orders them by how many of the field's name words (msi-product-code-guid -> msi, product, code; the FORMAT_WORDS dropped) are among the CONTEXT_WORDS = 2 camel-case words before the token ({DD1A80D5-...} after ProductCode goes first), file order after; each token as stored then its written forms (7.6). a 32 / 40 / 64-hex token goes once, as stored, never its case or 0x forms (as_stored), and a field whose question asks a hash to compute ("compute sha-256 over ...", asks_digest) gets only a token that is the md5 / sha1 / sha256 of one of the set's files (member_digests: a manifest that stores its members' digests).
  3. assembled: the pieces of the set's small text / json members (string values, word runs) joined two or three in any order that fullmatch the asked shape (ANS-7 + E37D13 + 4754DEA01).
  4. an enum of two or three values (autopilot.enum_values, ENUM_SWEEP = 3) sent as such, true first, when every reader missed, before the hand-off row (7.9).

the decoded blobs of a file set are cached (texts, the last _TEXTS_KEEP = 2 sets), so learn and sweep run once per field over the same set; learn returns at once for a field with no token_pattern and skips the ranked scan when no written form of the graded value is in any blob. the sweep reads its stop before every field (sweep_stop), so a fixer's save never waits behind a long derivation.

the learners (autopilot.learn_gaps, in follow). when the single is won (by anyone, in any form) and solve.py does not reproduce it, each field solve.py cannot read gets a rule by field kind (locate.kind, shared across tiers), the learners tried in this order under one LEARN_BUDGET_S = 20s budget over every field (learn_gaps logs learning N unread field(s) over M member(s) and learn capped at 20s before <field>, so a lane's silence after a hand win is never unexplained):

a field no learner explains logs no located solver: <field> is no stored token, role route, reader output, codec chain or header integer, and the agents take it.

the located solver (autopilot.located_solver). write_solver writes files/located/{solve.py, learned.json}, a solve_set that calls locate.solve (solve): the lane's solve.py first (the ruled fields together, then one by one past a crash; a set it cannot answer split per field, as dryrun does), since a mastery's sets can hold variants the single never showed that the lane solver reads and no rule would; a learned rule answers only what the lane still cannot, dispatched by locate.answer (an inventory rule to inventory.answer, int_at to the header integer, role to routed, else located). it is used only when solve.py itself does not reproduce, and only if the located solver reproduces the graded single (preflight checks it again). its per-field pass (unread) skips a one-field call the reproduce check just saw raise (autopilot._recording; a timeout is never taken as a crash). the located solver is a real file in its own directory: its code stamp and read lease cover itself + locate (+ runner), and a run saved with it replays (uv run replay) as long as files/located/ is there.

limits. a token split across ole2 sectors is found in the stream, any other container's fragmentation is not; a value that is computed (a hash of the file) or written in another form than stored is no stored token; the learned words must precede the value in the bytes (a value before its name, a table cell by column, is not learned).

the hand gates (gm.py). gm.rule0_gate: a lane fixer's gm.py guess / probe on its own single is refused until the challenge has handlers.py (--hand for a value no code read, logged and mailed to the parent); the intel --value path too. hand_cap: anyone else (a parent, a helper, a second-runner assist, another lane's fixer) probing a lane single whose fixer is live stops at one accepted field: the next is refused (exit 4, fixer <name> is live: mail the value ... or the recipe (coord.py patch), logged HANDCAP); autopilot, the rerun, a lane with no live fixer and web pass. only the fixer's handler reproduces a hand win; every hand-won field without one waits on CHAIN WAITS ON solve.py.

drill (tools/comp/rehearse.py located_drill, part of autopilot.py --check every night): for every lane's newest fixtures, each field whose graded value is a token stored in the single's bytes (stored_fields, rehearse.py) loses its reader, and autopilot alone must flag all three tiers on the fake gm (sweep -> located solver -> chain). the line rehearse located-drill <lane> ok (N stored-token field(s) ...; single in K submit(s) ...) is the coverage; a lane with no such field prints nothing.

tests: tests/test_locate.py (token shapes, an msi's bytes sweep the productcode first, learned words on fresh utf-16 bytes else a positional unsure, a bare token learned by position, the located solver keeps the lane solver's answers, autopilot reproduces the graded single end to end, the drill flags every tier), tests/test_located_solver_path.py.

7.8 race fixes in one edit (hlib.fields)

one place and one shape for a fix in every lane:

7.9 reader outputs and the hand-off (hlib.inventory)

a field no lane reader names is often a value a reader already outputs under other words (winart.lnk target for lnk-target-path-windows, office.info external for ooxml-external-relationship-targets). autopilot tries such a field with no agent while the fixer works it, and a field it cannot get fails with what a fixer solves from.

7.10 crash fields and the scratch guard (runner, solved_guard, toolcheck)

7.11 question recipes (hlib.recipe)

a binary question that spells its own rule ("true exactly when the final bytes contain nimstring or nimmain", "count names that match /^_d|\b_d_/", "read elf e_shnum", "join with one lf byte and no trailing lf, then sha-256", "render header byte 5 as 0x plus exactly two lowercase hexadecimal digits", "classify ... submit one listed lowercase token") is read by recipe.parse(question) into a Recipe(select, op), from those sentence shapes and nothing wider:

7.12 shared plumbing: ids, archives, formats, fixes, tables, caches and memos

7.13 warmups: the line decoder (decode)

hlib/decode.py turns a warmups blob (numbered NN<tab>chunk lines, each its own codec, or one text under a stack of codecs) into the slug its last decoded line names. entry points: find_slug(raw, pattern), slug_candidates, slug_answer, peel(text, goal), mixed_lines / lines_decode, slug_pattern(question).

the day table. numbered line k carries day k's codec in every tier. DAY_CODECS holds one row per day: b64, rot13, hex, bin, dec, oct, url, escapes, jsonstr, b32, b64url, reverse, atbash, rot19, rot47, morse>hex, baconhex, railfence3, utf16lehex, utf16behex, b64inflate, b64inflate, a85, netstring, uniesc, brainfuck, ook, reverse>hex, rot13>b64, url>url. a row may chain steps (a>b = a then b, run by day_codec); a transposition row carries its width (railfence3, columnar4, route4: fixed_width), never a keyless family that would read every line by whichever width ranked first. day_table is DAY_CODECS, then LEARNED_CODECS (a chain autopilot learns from a graded single its solve.py did not reproduce, decode.learn / locate.learn_codec; decode.remember("a>b>c") is the one-line fixer override), then PEELED_CODECS (the last confident chain find_slug peeled or chain-searched in this process; ungraded, so last: the chained time trial's fresh blob and a mastery's new line read by it with no learn). _positional reads each line by its row and searches the one line past the table with the rest pinned (_new_line); a learned or peeled chain that does not read it falls back to DAY_CODECS alone. a line read by anything but its row (a guessed one too) is noted in hlib.tables (decode.DAY_CODECS, 8.3) and fails replay --tables. line_report, the crash text, leads with the lines the table did not read, raw bytes beside each, then the worst-reading line.

the new line. _new_line tries tiers in cost order; the first tier with a reading that ends the message in a known slug wins, ranked by most named codecs, then hint words, then graded slug words, then english (_hint_rank): what the prompt names (_hint_step, where a named cipher's identity reading stays: bifid on a repeated-letter chunk), chains of the hinted codecs, one step (LINE_ORDER), the line as written, the transforms LINE_ORDER leaves out (_left_out), the keyed step, two steps, and last chains of every day-table step. a reading whose slug is all graded words returns at once (slug_vocabulary: every word of every graded warmups slug, from tests/fixtures/day*/warmups-day-*/expected.json). hints are the challenge's own text: HINT_FAMILIES maps each codec family to its prompt words (every transform has some; a stem matches, _says: "inflation" names b64inflate, whose family is compress / inflate / deflate / zlib / pressure / wrapper and every wrapper's name; utf16be (big one, big entrance, big-endian) is split from utf16le, the generic two bytes / little one / endian words; utf32 takes four bytes / ucs-4); SHIFT_WORDS reads a shift given in words ("a week off" = 7, both directions); REPEAT_WORDS ("twice", "double") name a codec applied twice; hint_codecs / named_units list what a prompt names (the caesar family, the xor searches and the many-reading readers are never chain units).

the codec inventory (TRANSFORMS, name to reader; readings(name, text) gives a reader's outputs, several for ALPHABET_READINGS / XOR_SOURCES):

line order and shape gates. LINE_ORDER is the sequence the beam and the one-step tier try on a line, cheapest and likeliest first. SEARCHES (the xor readers and the 24 shifts) run only once the plain codecs found no dictionary slug, except a shift the prompt names (_HINTED); the many-reading readers (route, columnar, bifid ...) and the whole-text shifts are never in it. NON_ASCII_ONLY codecs are skipped with one isascii() on an ascii chunk; LOOKS_LIKE gates netstring, bencode, hexdump, roman, numberwords and rle by a cheap shape regex (each otherwise costs ~0.08s of a 48-set mastery's budget in call-and-exception overhead); SAME_FAMILY stops a second step that only repeats the first.

the chain search. chain_search is a breadth-first search over chains of up to CHAIN_DEPTH 3 units (chain_units: the prompt's named codecs first, CHAIN_HINTED 6 at most, then every day-table step, then each chained row as one unit) inside CHAIN_BUDGET_S 1s; a reading seen before is not read again, a unit of the last unit's family is skipped. find_slug runs it on a whole blob between the single transforms (every reader alone, less the many-reading ones) and the peel, accepting only a slug of dictionary words (_chain_goal); the winning chain goes to PEELED_CODECS. the same search is _new_line's hinted and last tiers.

the peel and its budget. peel(raw, goal) is a bounded breadth-first search over every TRANSFORMS reader: the whole-text shifts and unstripped whitespace once, then depth 8 at most, PEEL_NODES 20000 readings, PEEL_BUDGET_S 8s of wall clock per blob (never a per-set pool: a mastery peels ~138), checked between readers too so one node of a huge text cannot outrun it. peel(early=True) (find_slug's goal) meets the goal as each reading is made; every other caller checks at pop time. out of budget with slug-shaped readings in hand (near), it raises PeelBudget carrying them, dictionary slugs first, and slug_answer sends them as low confidence instead of losing the set.

the line beam. mixed_lines sorts numbered lines, lines_decode takes any line list; both read the message as one hypothesis: _beam joins every reading of every line left to right and scores the join by letter-pair fluency (the committed lexicon hlib/lexicon/: words.txt.gz, the system word list plus every graded and harvested slug word, and bigrams.json, letter-pair log-probabilities), BEAM 64 hypotheses, PER_STATE 6 per last character (after the last line, per reading of it: every message ends in the slug's digits). _pick prefers a slug of dictionary words, then one after a space or colon, then fluency. the order in _lines_decode: the positional table, the budgeted unread line, the beam, a 4x wider beam, the beam with searches, _repair (one- and two-step readings of each line under the slug; named codecs rank over searched ones, _searched, so a rot-n search never wins confidently over a named reading), _keyed_repair (keys from the prompt and the other lines), the per-line best-first combination search (_first_combo, a heap over COMBOS 5000 index vectors, each reading scored once by english), two-codec lines only once every single-layer combination has failed, and a depth-2 beam last.

slug finding and shape. slug_pattern(question) builds the slug regex from the question's syntax (word and digit counts in words; SLUG, three words and two digits, when it says nothing parseable). slug_hits / slug_prefix count a slug that runs past the asked shape (the shape plus -token groups, _extended) as found, so a verified line reading is never traded for one that ends the shape; slug_answer returns the whole token, then its asked-shape prefix, as a confident pair (the grader takes the whole; an unsure pair would fail autopilot.fixed_run on every mastery holding one). find_slug's order: a learned or peeled chain on the whole blob, numbered mixed lines, per-line b64 / rot13 (decode_lines), lines in file order, the chain search, the peel. slug_candidates returns at once when find_slug's slug is all dictionary words (the beam's backups serve only a rejection and were most of a mastery's solve); otherwise it adds the next distinct slugs the beam ranked, SLUG_CANDIDATES 4 per field, retried by the runner inside the attempt.

confidence. SlugAnswer.confident holds only for a slug of dictionary words (_known_slug); another slug carries the per-line report. english counts the message's stock words (ENGLISH). a mastery line past the table that nothing reads gets UNREAD_LINE_BUDGET_S 0.03s of one- and two-step readings, then keys in batches of UNREAD_KEYS 8; only a codec's reading is confident, else LineUnread goes out with its slug-shaped readings (a search's or key's dictionary slug first) and the set is a runner.Unsure whose report names that line only (no slug-shaped reading: the report is the crash). a new line read only by a keyed or searched step (GUESS_STEPS; the day table's steps and a shift the prompt gives by number are codecs, not searches) is confident only with GUESS_LETTERS 7 letters in distinct words (_word_letters); fewer and it leads an unsure list (LineGuess, carried through find_slug as Guessed so a race fix wrapping it keeps the doubt).

speed. the same nodes and readings at c speed: the xor readers xor through bytes.translate tables (XOR_TABLES), rot-n / atbash / rot47 / rot5 through str.translate, the base-n readers fold digits by table and halves, rail positions (_rail_positions) and letter-pair costs (_char_cost) are cached, hint parsing is cached per prompt, the lexicon loads once.

tests. tests/test_warmups_chains.py (every ordered pair of day-table units as the single and as a mastery's new line, sampled triples, the budgets, the guess rules, the escapes layers); tests/test_day23_decode_rows.py through tests/test_day30_decode_rows.py (each row and its siblings); tests/test_tables.py (the missing-row note); tests/test_capability_contract.py with tests/fixtures/capabilities/transforms.json (a round-trip sample for every TRANSFORMS name, hint words for every family); tests/test_mutations.py; tests/test_timeouts.py (a compression bomb); the warmups fixtures under tests/fixtures/dayNN/warmups-day-*/.

7.14 metadata: records and properties (forensics, records, containers, headers, composites, msi, synthesis, media, netcap, winart, office, mail, pdf, hive, winlogs)

a metadata field is <member stem>-<property>. the lane solver splits it (split_field: a dNN-<family>-NNN stem ends at its number, other stems lose the longest known property suffix, a pe version-resource name outranks a shorter suffix inside it, an unknown property is still split after the stem's last number so the guess below runs), answers a synthesis field first, and hands everything else to forensics.property_value(prop, data, name, question=) with the asked question object (the question= keyword reaches record_guess).

how a property is answered (property_value, in order):

  1. exact_value, what the lane answers by the property's own name (none when only a guess is left): magic-prefix-hex-N (magic_hex); the PROPERTIES registry (digests, file_format, text and bom facts, jpeg_sof_dimensions, image_size, the png chunk walk png_info behind PNG_PROPERTIES, PDF_PROPERTIES with pdf-header-version the graded one, the zip census and provenance checks, the pe properties and the ranked executable labels); relation_value (forensics.relation_value, below); every msi- property (msi.value); a struct name (struct_named: a widthed tail, or a format word under a known family) read by struct_value or a crash, never a word guess; a pe comparison and a version-resource name; a family-prefixed generic (<kind>-file-size-bytes, the digests, <family>-file-name); png-<type>-chunk-count; pdf-xref-offset; _media_value (below); last the FALLBACKS regexes for unknown *-dimensions-pixels / *-width / *-height and the png passenger / after-iend / crc and pdf version spellings (a fallback raising valueerror on a family with a record of its own returns none, so the record answers).
  2. CROSS_LANE, the hook hlib.crosslane installs (7.5): another lane's reader of exactly this name.
  3. the guessed value: embedded moves to a sqlite / pdf magic past offset 0; a RECORD_FIRST family (sqlite, mft, png: their records keep ranked forms the header parse lacks, and the mft header guess read record.hard_link_count for "record count") tries record_guess first; then headers.guess; on its miss a STRUCT_FIRST family (elf, pe, macho, macho-fat, wasm) tries bare_struct_value, so a header struct keeps its ranked forms ahead of the record; then record_guess; then exif_guess: only an exiftool tag covering every word of the property (with synonyms), a tie never, an exiftool exit on an unknown container a miss like any other. a full miss is one keyerror carrying every miss and the closest struct fields.
  4. last resort bare_struct_value: a struct field named with no width or format word (elf-e-phnum, elf-ei-data; every graded header property carries one, elf-e-machine-u16-hex) answers as the decimal value then the padded hex.

the struct tables (STRUCTS: 39 header structs over the executables with mach-o's fat_arch, dex, java class, wasm, neko (for recipe.HeaderField, 7.11), the image, archive and media headers, dmg koly, minidump, ttf, ico and an ntfs file record; aliases such as mscab, mdmp, ntfs-file-record, trailer): <family>-<struct>-<field>-<width>-<format> names (the graded <format>-<struct>-<field>-<width>-<fmt> shape), width-less and struct-less names resolved when one field fits; struct_field(prop, data) settles a field two structs of a family name (header.cputype / fat_arch.cputype) by the struct whose records these bytes read; a miss lists the closest fields. struct_alias and version_alias are race-time one-liners from solve.py registering a spelling as a PROPERTIES reader. a trailing set / list / values / all word (struct_set_value) reads the field over every record of the one struct the bytes carry: a compact json array of the width-padded hex, ranked sorted-unique, record order, then decimal (macho-cpu-type-set graded ["0x0100000c"]).

headers and composites: headers.PARSERS are raw <struct>.<field> readers for gzip, tar, bmp, gif, png, riff (webp vp8x included), sqlite, id3, zip, pdf, 7z, cab, dmg (koly tail, a negative magic offset), minidump and mft; guess(prop, data) matches an unknown property's words to them (every word covered, the last word shared, a tie never). composites.derive / with_composites add the keys a question would name from the keys a flat record has: a width and a height under one stem give <stem>_dimensions / _resolution (WxH) and _aspect_ratio, a _ms gives _seconds, the record's key style kept, a record's own key never overridden; derive runs inside headers.guess, forensics.record_guess, _header_guess, forensics.exif_guess (the media nodes keep their own table) and inventory.with_composites (7.9).

sniff and the record registries. forensics.sniff names an artifact from bytes only: cafebabe_kind (java class vs fat mach-o, like libmagic), riff wav / avi, an ftyp box as mp4, zip_kind for any PK (apk, nupkg, vsix, epub, odf, ooxml, wheel, jar, webext, else zip; marker files read only when small), prefetch, thumbcache, a magic table of some fifty signatures (crashdump for pagedu64 / pagedump, lime staying memory), then containers.kind after every magic (only where records.kind agrees, so a fixed vhd or a partitioned image keep theirs), a dmg koly tail, tar, ese, pyc, eml, text, binary. records.kind refines text / binary into plist, iso, vhd / vhdx, disk, a container kind and the small formats.

every family hlib parses is a record family: RECORD_READERS in forensics (lnk, prefetch, eml, ooxml, apk = android.apk_record, jar, java-class, elf, pe with its managed columns from dotnet.pe_record, macho / macho-fat, wasm, pdf; _lazy_record imports the module at first use, android before java) and records.READERS in hlib.records (ole2, evtx, registry-hive, pcap / pcapng, sqlite, plist, iso, vhd / vhdx, disk, torrent, x509, ini, rar, ttf / otf, ico / cur, rtf, ese, tar, tiff, dex, the zip kinds through records.zip_record, pyc, thumbcache, wim, hlib.containers' nine readers, mft, pdf, png, 7z, journal, pst). forensics._record_reader answers a kind both tables read (pdf) from both, the family module's key winning. RECORD_HEADERS (evtx -> records.evtx_header) is the header record of a family whose whole record is slow: _header_guess answers a property naming exactly one header key, the whole read everything else.

record_guess answers the field whose every word the property names in one spelling (RECORD_CANON: modified = write, description = name, volume = drive, id = guid), most words wins, a tie never, a leftover word another field owns never (a crash, not a wrong answer); the property's family word and RECORD_FAMILY's (lnk: link / shortcut / shell; sqlite: database / db / table; disk: guid partition table / image / img; containers.FAMILY) name the family, never a field; qualifiers shape the value (hex -> 0x.., an address out of <..>), symbolic fields and times go out ranked, an empty field is none; an unreadable record is a miss. tests/test_record_grid.py (every record key answers as a property) and the coverage contract (every sniff kind has a route) guard the registries.

format readers by family:

media and its tool wrappers (media): media._tool runs a tool on a temp file under a 5s timeout (TOOL_TIMEOUT_S, under a timed set's clock) and turns a missing / hung / failed tool into a clear runtimeerror. media.exif_text (exiftool -args -n, exact text, one run per distinct member and suffix, EXIF_KEPT 256) backs forensics.exif_guess. media.probe / primary_stream / stream_count (ffprobe streams + format; probe(timeout=)) back media-primary-<audio|video|subtitle>-<key>, media-container-<key> and media-<kind>-stream-count (graded pcm_s16le); primary_stream takes the default-disposition stream first, cover art never. stream_value / container_value: ffprobe's key, else a composite (STREAM_COMPOSITES: dimensions = width x height (graded 320x180), the coded / display / rotated dimensions, rotation, orientation, frame rate, aspect ratio, duration, codec, pixel format, frame count, coded_padding_present (a fact; a crop verdict is no composite); CONTAINER_COMPOSITES: duration, size, major brand, title, stream count), tags and side data flattened in, else every key whose words cover the asked ones (ranked), else a keyerror naming the keys the node has (forensics._media_value wraps it with the property). media.qr reads with zbarimg, else opencv's QRCodeDetector. the relation keys (below): pixel_key, image_print and similar, crop_thumb / crop_score / crops, audio_keys / video_keys (one ffmpeg run capped at DECODE_S 2s). every media memo is a Kept keyed by the member's sha256, never its bytes (7.12).

captures (netcap): netcap.hosts(data, timeout) is one tshark run (ip / ipv6 ends, dns query names, tls sni, http hosts, each in first-seen order, kept by the capture's sha256; a capture cut short answers its whole packets; nothing read or a run past its timeout is a miss). records.pcap_network (hosts, dns_names, tls_sni, http_hosts, PCAP_NETWORK_S 3s, over netcap.hosts) joins a pcap / pcapng record in forensics.record_guess only when the property names a PCAP_NETWORK_WORDS word (host, dns, sni, tls, http, queries, domains, server, peers, endpoints; never names: interface-names runs no tshark), with PCAP_NETWORK_ALIASES beside the four lists (dns_queries / domains, sni_names / server_names, servers / peers / endpoints); pcap-packet-count and the inventory path never run pcap_network. the other wrappers (usb keystrokes, icmp payloads, http object export, a packet summary) serve the solvers and uv run triage.

installer and synthesis fields. msi: stream_name (the msi table-stream name encoding), properties (the property table through the string pool, 2/3-byte refs), summary (summaryinformation: package code = revision number), value for any msi-<property words>[-guid], always the stored text. synthesis: a cross-member field the question spells out (...-synthesis-sha256-order): the named files, a per-file property (md5..sha512, crc32, size, or any property the lane reads) and the order or the low / high end.

prepared readers (winart, office, mail, pdf, hive, winlogs): winart also carries memory-image kinds and a volatility wrapper, pyc version and decompile; office the ooxml macro / info reader and repair_zip; mail eml and .msg parsing, urls and mail.attachment_members (an eml's parts, an outlook .msg's PR_ATTACH_DATA_BIN; none for an msi or a .doc); pdf the object walk behind pdf.record; hive a hive walk with persistence and userassist views; winlogs evtx events with counts, script blocks, processes, services and logons.

disk and container records

the relation family and its semantic keys

7.15 mobile: apk, dex, android

the artifact is an apk (a time trial's extracted members, a mastery's bundle of apks) whose assets/huntress_puzzle.json (PUZZLE; a moved asset is the one json member named like a puzzle) carries a codec, a data and usually a key; the answer is an ANS-[0-9A-F]{16} code (ANSWER; _answer unwraps quotes, brackets and a json byte list). apk.answers splits the asked fields by their format: an ans- field goes to apk.from_files, any other to android.answers by exact name, never the code under an app field. apk.is_apk wants a compiled manifest and a payload (dex, resources.arsc, res/ or lib/); dex.is_apk_zip counts a manifest-only split as a package.

the codec chain (apk.decode(puzzle, keys)): the codec text is split on CHAIN separators, filler words (FILLER) dropped, each word mapped through CODEC_ALIASES to a step. text codecs run through the warmups table (decode.TRANSFORMS, 7.13); the byte steps (BYTE_STEPS) are the keyed ops KEYED (xor, add, sub; caesar / rot are add), rc4, vigenere, inflate (zlib / gzip / deflate), aes (AES_STEPS: the mode from the name or all four), BLOCK_STEPS (chacha20, des3, blowfish: the same key derivation cut to each cipher's widths, nonce / iv from a field, the data's head or zeros, modes from the name or cbc then ecb, pkcs7 at 8), fernet, rsa, ecies; TEXT_KEYED holds the transpositions and substitutions the puzzle keys. a bare keyed codec implies the base64 layer. a key is read as hex, base64 or text, or the sha256 / md5 of a passphrase (key_readings), or derived by the kdf the json names (_kdf_keys): pbkdf2-hmac sha256 / sha1 / sha512 with its salt and iterations, scrypt with n / r / p, argon2 id / i / d (_argon2: the json's time / memory / parallelism or five usual profiles). fernet is the token's aes-128-cbc body under the 32-byte url-safe base64 key, the hmac unchecked; rsa reads oaep sha1 / sha256, pkcs1 v1.5 and raw under a pem from a json field or a member (app_pems); ecies takes the ephemeral public key at the head, eciespy's hkdf over ephemeral + shared point or the ecdh-x variants, aes-256-gcm, _ec_mul for the shared point. a json mac is a verdict, never a block: the reading whose plaintext or data verifies under the key wins (_mac_verdict). a puzzle with no data, or whose data names an image, takes the qr codes (media.qr) and exif text of every image under res/ and assets/ (app_codes). a key the json lacks or lies about comes from the app's own strings: app_keys over dex / arsc / json / xml / properties members and lib/*.so, runs beside a key-ish word first, APP_KEYS 400 at most; member_keys the same over extracted members.

split, indexed, packet, table:

when the chain yields no code, decode tries in order: the chain under each of the app's strings as the key (no key field in the json); _keyed_peel (the table step, apk._table_readings; every single-byte key, the 2-4 byte keys the ans- prefix fixes, then the index key schedules apk._schedules: a + b*i and a ^ b*i fitted from the prefix on bytes 0-1 and checked on 2-3, of every keyed op over the data's byte readings, forwards and reversed, so a renamed or unkeyed variant of the table family answers); the layer peel below; decode.peel (7.13); last answer_scan: every one-step reading of data and its reverse searched for the code shape, then the same keyed by up to SCAN_KEYS 64 of the app's strings, then every first-step reading read once more (SCAN_READINGS 200, SCAN_SECONDS 2.0), then the json's raw text; one code answers, several come back ranked by how many readings hold them as AnswerCandidates (apk.AnswerCandidates, a ValueError with .codes; decode_puzzle / from_files hand them to the runner as a ranked list). a search that answers notes the codec in tables.note (8.3).

members, bundles and parts (apk.from_files): a data that names a member of the apk (MEMBER_PATH; assets/payload.dex under codec dynamic, never a base64 blob) is read by path or base name and the one ans- string stored in it is the code (member_code; several is a crash, none falls through to the chain). a codec naming the analysis (ANALYSIS) or a json with no data goes to the interpreter (computed_code -> dex.answer_codes, the question's "in package x"). a set holding several android packages (an .apks: base.apk + feature.apk) is one app: _base_apk picks base.apk, else the apk with code; every apk's members come into view under <apk>/<member> (dex.apk_members), and a computed puzzle of a bundle goes to the interpreter with the whole set. apk.with_feature_parts fills a split puzzle's empty part (parts ["", "ANS-7", "E37D13"], order [2, 0, 1]) from any json member named feature / split / config / part carrying {"part": ..., "part_index": N} (part_index and its spellings, PART_INDEX; an index past the list appends), e.g. 4754DEA01 in feature.apk/assets/huntress_feature.json, before the usual join. _decoded_or_computed decides between the data and the dex: data a step or search reads answers; data nothing reads goes to dex.answer_codes before the decode error is raised (a dex with no ans- method leaves the error standing); a code the dex computes beside readable data ranks second (_and_the_dex, behind a cheap check for the ANS- literal in a dex's bytes).

the layer peel

apk._peel_layers reads a stack the codec name does not spell (_peel_layers, called by decode with the data and the puzzle). it runs breadth first over the keyless layers to PEEL_DEPTH 5: base64 / base64url / hex / base32 read strictly, rot13, rot47, reverse, the UNKEYED byte steps (nibble and pair swaps, bit rotations rol1-rol7 and not; ror k is rol 8-k, rol4 the nibble swap; the same names read in a chain), zlib / gzip behind its header. a run of bit and position steps is read in one order only (_redundant; _bit_viable prechecks a rotation). the unstripped data is a second root when it differs from data.strip() (raw bytes stored as latin-1 text whose outer byte layer put a whitespace code on an edge byte); a _json_code (the one ANS- in a json wrapper's string values, a sentence or split parts) reads in the peel and the keyed search.

a keyed layer (xor / add / sub; a fitted sub reads as add under the negated key) is fitted, never searched: the code's 4 known bytes carried up through every tail of up to TEMPLATE_TAILS 3 keyless encodings (_templates: the 16 one-digit fillings of the code's body, the bytes all 16 agree on, grouped by length) give the key a fixed, walking (a + b*i, a ^ b*i, the _schedules forms) or 2-4 byte repeating key must have, checked on the other known bytes (the integer loop _keyed_raw); the template's steps undo it. a compressed layer's head through up to 2 tails (_heads) fits a key over an inflate. the json's key is a hint for the first keyed layer only: as a repeating key and as k_0 of a walk of step ±1..±3, kept only while the output is text or a stream (_plausible). PEEL_KEYED 2 keyed layers fit in one stack: an outer key over a text layer has no known bytes, so it is read as the keys whose undo is a text layer a keyless step consumes (_text_keys: one 256-bit mask of valid k_0 values per walk, steps up to ±16), fitted at the outer position over binary data only; the second layer's template and deflate searches run only under a fitted first key on a text decode's output (the example stack: xor key=36 > b64 > add key=schedule 7, then 10, 13, ...). an unnamed block step under the json's key (BLOCK_LAYERS: aes 16, des3 8, blowfish 8, the key read as the named chains read it): cbc with the json's iv or the data's head, then ecb, only with a key, on a binary ciphertext surface and for the first keyed layer (a zero iv self-syncs past a bad block into a near-miss).

a reading only a repeating key gives is weak (a key a nibble off maps hex digits to hex digits) and waits for the search to end; the keyed search's 2-4 byte prefix key the same. a strong reading opens one more level unless nothing can beat it; then the fewest bit steps beside the key win (_bit_sides), then the shorter walk (_key_cost), then fewer layers. when both placements of a stack decode (a rotation above vs below a walking key: rol6 + rol6 reads as rol5 + rol7 around the key) both go out as ranked candidates through apk.AnswerCandidates: exact rank ties (_tied_codes) and the family rule (_placement_codes: the same key op and the same non-bit steps); the runner bisects inside the attempt. caps: PEEL_SECONDS 1.0 and PEEL_NODES 20000 per set inside apk.decode; a set whose codec is analysis never runs it. tests/test_apk_layers.py holds the matrix (2-, 3- and 4-layer stacks under the true codec name, a lying one and none, with the true key, a lying key and none; full_matrix() adds noise; the rot matrix ranks the rotation placements; the whitespace edge cases), tests/test_apk_twokeyed.py the two-keyed and block layers, tests/test_apk_table.py the table family under any step, key or name, tests/test_readiness.py DAY29_PUZZLES the family at release.

the dex interpreter (dex): program(b) parses a bare dex or an apk's classes*.dex in load order and keeps the app's own classes (the launcher activity's package, a bare dex's main() package) in java's view (7.17): members, code strings and invokes, annotations; listing / disassemble (LISTED 40 methods) feed triage (uv run triage <apk> prints the puzzle, the ANS- methods and what they call disassembled, the native methods with their jni symbols and the lib/ exports). Vm runs dalvik: registers hold ints (a float as its bits), str, none and the object kinds (Obj, Array, Named, Stream, ...); every opcode but invoke-polymorphic / custom and method handles; <clinit> runs first; STEPS 1_000_000 instructions per candidate method, DEPTH 200 nested calls; a throw, a thread, reflection on a class the app lacks or a call outside the framework layer raises ValueError naming it, never a guess. dex.answer_codes(app, package) runs every own string method holding the ANS- literal, the ones another method calls first; a called method's failure is the cause, never a decoy's code; codes come back ranked.

android (android): manifest(b) reads the binary axml into a Manifest (package, versions, sdks, permissions, debuggable, allow_backup, components with intent filters, deep links, providers, meta-data, the cleartext attribute and the nsc reference); dex(b) an apk's dex strings / classes / methods; info(apk) an ApkInfo with the signing certs and every url. network_config reads a nsc (compiled or plain): nested domain-configs inherit from their parent, top-level ones from base-config, trust anchors, pin-sets, debug-overrides. transport builds the Transport: every nsc variant, the referenced one (by @xml/ name or through resources.arsc), the effective cleartext (a referenced nsc replaces the attribute on api 24+), user-ca trust, the cleartext domains, a status and the manifest vs nsc disagreements as notes. app_fields answers an app question by exact name (cleartext, nsc, provider, debuggable, min_sdk, target_sdk, ...) plus manifest_fields (package_name, version_name, version_code, permissions, main_activity, the component counts) plus a race-time FIELDS one-liner (android.FIELDS["x"] = fn over the manifest and transport); signer_fields adds signer_sha256 / _subject / _issuer / _count from the first signing block (openssl; no block, no openssl or a run past its cap: no signer fields, every other reading stands). app_values(apk, timeout=) is the read-only mapping of all of them plus dex_count / file_count, cached once per apk per set (setcache.on_bytes); its signer_* keys run openssl only when a signer key or the whole mapping is asked (AppValues); timeout is each openssl run's cap (proc.FAST_S by default), a capped caller gets AppValues.capped(t), a view with its own cap over the set's shared values, and a signer read cut short by its cap is never kept. through it crosslane (7.5), the inventory sweep (7.9), the located learner (7.7) and the metadata record reader (apk_record, forensics.RECORD_READERS["apk"], 7.14) all see apk.package_name and its siblings; android.answers reads a field from app_values, else another lane's reader of that name (crosslane), and crashes on an unknown or empty one. _certs keeps a signing block's cert by the block's sha256 in _CERTS, a media.Kept(CERTS_KEPT) of 32 (7.12), so equal blocks under other names or in other sets never rerun openssl, whatever the timeout; a run that raises is never kept. tests/test_android_manifest.py, tests/test_apk_aes.py, tests/test_apk_scan.py, tests/test_apk_analysis.py.

7.16 script: specimens and descriptors (specimen, script, vbscript, batch, python, wmi, carriers, graded)

every script specimen hides a descriptor: v=N|steps=a,b|k=v... or a json {"steps":[...],"adapter":..,"values":{..},"stageCount":n}. specimen.parse reads both (the json flattened) and accepts a text only when its steps are snake_case tokens. the graded fields are the descriptor's keys under every alias (KEY_ALIASES: host-id / host, file-path / path, note-name / note, stage-count...), the role and attack of its step graph, and per-language labels. the lane solver reads every member through carriers.to_text first, then the readers below.

the descriptor

specimen.find returns a Found (descriptor, representation, transform); find tries, in order:

  1. header_descriptor: the HUNTRESS_ANALYSIS_V1:<CODEC>-<payload> comment header (ANALYSIS_HEADER). HEADER_CODECS: frag (dotted hex), hb64 (hex of base64), hex, b64, b32, b85, lyr. an unknown codec name goes through every decoder, the payload reversed too; a header glued to its comment's closer loses COMMENT_TAIL; a header inside a host's string literal (the line ending in a literal \n) ends its payload at the escape, and a hex payload is its hex run. HUNTRESS_ANALYSIS_V1:SPLIT names no payload: the string literals of the next array (split_pieces) join to <CODEC>-<payload>, both orders, any joiner. HUNTRESS_ANALYSIS_V1:LYR-<layers>-<payload> names its layers _-joined (+, >, ,, | too) in encode order; layered undoes them last to first, then in the stated order, a reading counting only once it parses. LAYERS: b64, b32, hex, rev, frag, hb64, b85, zlib, gzip, deflate, bz2, xz; LAYER_ALIASES the spellings (base64, reverse, inflate, gz...); KEYED (xor, add, sub, caesar) take a hex key or a decimal shift (xor_5a, add_7) or recover it over every value, the v=N| prefix deciding. every step carries latin-1 text, so a byte layer survives the next undo; MAX_PEEL 4096 readings a payload may fan out to. a name no table undoes raises UnknownLayer; unknown_layers lists them.
  2. vbscript.descriptor: the Material_ / Codec_ convention (below).
  3. plaintext: the descriptor split over consecutive assignment lines (set "h0=v=1|steps=...", set v_a1=...) or whole in one literal.
  4. search: every literal (blobs: quoted strings, the quoted strings of one line and of a run continued over lines, long bare tokens, numeric lists, neighbouring lists concatenated, char codes) under every representation (vbscript.SEARCH_REPS, reversed base64, each header codec on a bare literal) x decoder (_decoders: identity, single-byte xor, add, rolling xor, rotate, repeating-key xor up to MAX_KEY 8; the key recovered from the known prefix, PREFIXES; the data reversed too, REVERSED labels). best = most fields, then the simplest codec, then the most graded step words.
  5. bare_runs, only when no literal reads: a headerless specimen with no quoted carrier (sed's s/^X$/<payload>/ glues a base64 payload to its / delimiters, which the base64 alphabet keeps). specimen.bare_runs yields each long token's / pieces, the token inside one / pair, and each header codec's alphabet run (CODEC_RUN: hex or dotted hex, base64) cut out of it; the v=N| crib decides.

a search that stood in for a header codec notes specimen.HEADER_CODECS (a lyr header's missing layer: specimen.LAYERS) in hlib.tables (8.3), so replay --tables fails until the codec has a reader. nothing found raises, naming any unknown layer.

answers(src, language), the entry for an unseen language: role (the descriptor's own, else script.role_of), attack, steps >-joined, representation and transform (a HUNTRESS-CODEC: comment header, else the finder's labels), every descriptor key under its aliases, every HUNTRESS-<WORD>: header (headers), a generic_control guess; our own labels travel as _guessed; wmi fields come on top (_with_wmi). language-specific labels (styles, expansion) stay gaps for the dry run to name. answers is also where the generic check routes a known language treated as unseen.

composite fields (composite(name, question, answers)): DERIVED (<key>-token after the first hyphen, <key>-host a url's host, <key>-leaf a path's last segment), indicator (the first key present in the question's stated order, INDICATOR_ORDER, else INDICATORS), and n labels joined by a separator (JOINED, LABEL_SPLIT, COUNTS, MAX_COMPOSITE 16), each phrase matched to a key by its stemmed words (phrase_keys, STEM_SUFFIXES). a phrase no key reads is a keyerror, never a guess.

the stage graph

stage_graph(texts): member name -> descriptor. a field is a relation only by its grammar (relation: a BEFORE / AFTER word before its kind; the kinds MEMBER_KINDS, language, count). related reads a relationship the descriptor does not state: claimed next stages, claimed parents or the members claiming it, their count or their languages (member_languages: EXTENSION_LANGUAGES, one family per LANGUAGE_FAMILY). impossible(g) ranks the members whose claimed family tree cannot hold (weight 3: a cycle, a parent in its own chain, a claim the other side contradicts; 2: a parent nobody ships, a language the stage is not; 1: two roots in one tree); impossible_answer(kind, g) returns them, their languages or their count, and [] when every claim holds (no answer, never guessed); anomaly(field) names the kind (ANOMALY words). a child the bundle does not ship is no impossibility (relay stages write their next stage): that is its own shape. three more comparisons over the same graph: unsupported (an adapter naming a network primitive, NETWORK_ADAPTER, while no step has remote), dangling (a claimed next stage absent from the bundle's files; shippedness is files, not descriptors, so ranked by how many dangle), fetched (a shipped member claimed as next by a stage whose own steps fetch). specimen.hinted(field, graph, shipped) routes by the field's words (unsupported / missing / absent / never there / delivered) and returns none for every other field (is_hinted); specimen.unsupported, specimen.dangling and specimen.fetched each return [] rather than guess. the lane solver dispatches hinted(field, graph, shipped) in its _impossible ahead of impossible_answer, the result a runner.Unsure. tests/test_specimen.py, tests/test_specimen_hints.py.

ranked candidates (best first; the runner submits the first and bisects the rest inside the attempt): labels (the question's enum, else the graded vocabulary of the field's suffix, ranked by words shared with the answers' evidence), unanswered (a relation, else labels), guessed (keys whose value is our guess: generic labels, the role of an ungraded graph, an attack that follows one or an ungraded language, any key an unconfirmed row disputes), unsure (keys no confirmed row backs while an unconfirmed one names a value: sent as runner.Unsure), alternatives (confirmed values, then unconfirmed rows, then blind ones; for attack the roles' techniques, an unlisted language ending with T1059 and T1059_SUBTECHNIQUES), ranked / unsure_note reading the _ranked.<key> / _note.<key> answer keys. vocabulary caches the label vocabulary on the stamp of its sources (label_sources: fixtures' expected.json, intel jsonl), re-stat'ed every VOCAB_TTL_S 2s, and the graded tables.

per-language answers

roles and attacks

role_of: ROLE_BY_STEPS (the graded table), a graph ending in handoff_script = staged_chain, else role_by_verbs (ROLE_VERBS, REMOTE / WRITE / DISPATCH words: four or more steps with remote and dispatch = staged_chain, then impact / uploader / relay / stager / persistence_descriptor / advanced_syntax by verb, downloader / stager / dropper / launcher / benign_utility by remote, write, dispatch), printed as a guess, costing the role field alone when wrong. role_from_choices against the question's enum: every graded role of the graph newest first, else the one choice sharing a word with the steps. attack_for(language, role): an interpreter role (INTERPRETER_ROLES) or an unknown one takes the language's t1059 sub-technique (INTERPRETER, else the parent T1059), any other its ATTACK_BY_ROLE. the tables load from graded through refresh() on a file-stamp change (file_stamp, tables_stamp), so a mid-race add reaches a long-lived process; the newest confirmed row per key is the default, earlier ones stay in ROLE_HISTORY / ATTACK_HISTORY, unconfirmed rows in ROLES_UNCONFIRMED / ATTACKS_UNCONFIRMED / INTERPRETERS_UNCONFIRMED; graded_roles lists every role a table names. tests/test_script_day23.py.

graded: hlib/graded/*.jsonl, one json row per line, append-only. TABLES: script-roles (key steps, a snake_case role), script-attacks (key role, attack T#### / T####.### / none, an interpreter flag), script-interpreters (key language). uv run graded add <table> field=value... --evidence <challenge>:<answer id> (repeatable) or --unconfirmed: verify wants each cited field's suffix in VERIFIED (-role / -attack) and its grader-confirmed value (confirmed intel or a fixture's expected.json, confirmed_values) equal to the row's (an interpreter role: any T1059 technique); the row goes out as one os.write on an O_APPEND fd, no lock; conflicts (an earlier row with another value) are printed and kept. a confirmed row is the default, --unconfirmed only a late, low-ranked candidate; rows reads complete lines only (an append in flight never crashes a reader); files_for(["script"]) is part of the solver's code stamp; graded list prints a table. tests/test_graded.py, tests/test_graded_lead.py.

wmi and carriers

wmi.model(src): a batch echo builder first (_reconstruct_echo: per target file, > truncates, >> appends; a mofcomp of every target sets Mof.compiled), then literal mof text (parse: a tolerant mofcomp tokenizer, #pragma namespace, aliases, object paths, DEFAULT_NAMESPACE), then powershell / wmic builders (build: set-wmiinstance, new-ciminstance, register-wmievent, register-cimindicationevent, [wmiclass], wmic; a variable resolves only as far as literal assignments go). the model holds Filter, Consumer and Binding records: a binding is resolved when both references find an instance and persistent when survival_problems is empty (a namespace in PERMANENT_NAMESPACES, a consumer class in STANDARD_CONSUMERS or one an __EventConsumerProviderRegistration names, a non-empty wql query; __temporary is a session-scoped subscription). every verdict is inference from wmi semantics, ungraded. trigger(query) labels a filter's query (TRIGGERS), decoded a -enc command. fields(mof) emits <qualifier>-<component>-<property> over PERSISTENT_WORDS, RESOLVED_WORDS, DECOY_WORDS and DANGLING_WORDS, the BARE aliases and counts; an empty negative group answers NONE (none, unsure); mof-compiled only from the echo path. describe prints one line per binding, the verdict and why. specimen._with_wmi merges them into a specimen's answers (the descriptor's own keys win; several values go out _ranked., a fallback _unsure with its _note.). tests/test_wmi.py.

carriers.to_text(data, name): a text specimen as is (is_text); FasdUAS through osadecompile; a upx-packed mz through upx -d on a copy (unpacked); AU3!EA06 (an exe's script resource, or a bare .a3x) through autoit-ripper (autoit_text); \x1bLua through luac -p -l -l (a listing only); else strings_text: every printable ascii / utf-16le run of MIN_STRING 6 as a quoted literal, also minus its first byte (a length-prefixed pool). a tool failure prints and to_text falls through to strings.

no specimen code runs on the host: jscript's recover functions run inside an empty vm with no activex / wsh objects, and every other reader is static (regexes, ast, listings and unpacks).

7.17 binary: compound and the format readers (elf, pe, dotnet, macho, wasm, java, native, swift, objc, trace, emulate, cil, versioninfo, ioc)

the lane's solver is compound.solve(files, fields, analyzers, questions). a dNN_ field names its day tag; the tag's members are the files whose name carries it (group keeps every dNN-tagged member whatever its format, and files under a tagged folder); non-dNN_ fields go to bare. a tag with no member, or no readable member, raises naming what it found.

how a dNN_ field is answered, in order:

the hcfg carriers. the config record (elf.Hcfg: version, key, crc32, the json, Hcfg.magic) is read from every member of the tag, main first, whatever its format, through the carrier stages (STAGES): raw, base64, hex, utf-16 wide and xor literals (FLAT_STAGES), a compressed stream (_compressed, INFLATE_LIMIT 64 mb), a packed overlay (_overlay), a jar's tail past its end-of-central-directory as a view; at most _CARRIERS_MAX 64 carriers a tag. every stage is magic-free: _any_b64 / _any_hex / _any_wide / _any_xor beside _any_magic, the any-magic scan inside _compressed / _overlay. elf.config_records (config_records(b, magic=b"HCFG" | None)) / crib_starts find every crc-valid record in the hcfg layout, under a named magic or any printable one (HGRF is the same 12-byte layout under another name), with its offset: the crib is byte[i+12] ^ byte[i+5] == '{', the crc32 over the decoded json decides. an appended record at a member's end is read by record_backward (recipe.record_backward: the named magic raw, then any carrier through RECORD_VIEWS, then any magic, the fallback logged). the question's magic word, member suffix (basename ends config.n), dotted json path and this_role compare go through recipe.RecordProperty (_record_property / _records_agree: the three .n graphs must agree with this_role omitted, else a ValueError naming the key and members). a tag hcfg two members share wins over a decoy. hcfg_storage is ranked by the search stage and the section the carrier sits in (base64 / raw_hcfg_section / raw_const_array; a fieldrva initialiser in a .net pe puts raw_const_array first; an ungraded carrier form is an unsure).

overlays and unpackers. a packed overlay is one more carrier: overlay finds an XXNN magic at the container's end (an elf's section table, a pe's last section; else the last magic whose u32le packed length at +8 ends the member, a jar's own tail included); the header gives k1 (+5), k2 (+6), k3 (+7) and the unpacked sha256 (+12), the payload starts at OVERLAY_HEADER 44. unpack tries the registered step (UNPACKERS, a tuple or a function; PK19 is xor k2, ror 3, xor k1), the step the question spelled, then a search over k1 / k2 / k3, the index, rotations, not, xor prev / xor chain: each candidate on the payload's first four bytes (HCFG, SEARCH_DEPTH 4) before any sha256, the full-hash search last (HASH_DEPTH 3, chains to CHAIN_DEPTH 8), keeping what the sha256 proves. the record inside answers config_*; overlay_offset / overlay_size / overlay_magic / unpacked_sha256 are producers, and overlay_fields answers offset / size / magic from the magic alone when no header rule reads; where a record sits in its carrier comes from hlib.placement (region_at(b, offset).name for the section, address_at for the virtual address, over elf / pe / mach-o regions) (magic_candidates, ranked: the container's end, the tag's day digits, the last past the container, the last at all). a step outside the search crashes naming the magic, keys, header bytes and whether the spelled step parsed or verified (overlay_note), with the one-line UNPACKERS[...] fix.

layouts from the question text. solve() notes every question's compute ... sentence and header offsets by magic (note_questions, step_from_text, layout_from_text -> STEP_HINTS, LAYOUTS; the magic search memoized). LAYOUTS is never seeded, the question fills it: Layout.scratch is a u16le scratch length (_overlay_at accepts a member that ends after the scratch, Overlay.scratch carries the bytes; the RS20 layout), the resource / scratch words are LAYOUT_WORDS (a gap stops at a comma) and LAYOUT_SHA_RANGE reads "+16..+47 are the scratch sha-256"; producers hcfg_resource_size / _length / _offset, scratch_size / _length / _sha256.

the format readers.

binary tools resolve to gnu binutils: homebrew's copy, then a g-prefixed install, then PATH. macos's llvm objdump/nm print a different format, so they are avoided on the laptop; on the remote linux vm PATH is gnu.

the dynamic trace and emulation.

fixtures: a nekoc build (brew "neko") embedded in tests/test_day21_prep.py, a config.n tail in tests/fixtures/carriers/day30.

technique producers

the technique producers live in compound and read through hlib.objc and recipe; every one reads the member's real format and crashes naming what it looked for, never a guess; recipe.SOURCE_WORDS makes a spelled question ("count the objective-c selector names in the __objc_selectors section") read the same lists.

tests: tests/test_binary_techniques.py, tests/test_compound_pair.py.

8. replay and fixtures

8.1 run / fixture format

<run or fixture dir>/
  manifest.json     {challenge, solver, artifacts: [{file, set_number, filename, kind}], expected_source?}
  assignment.json   the start/retry response (answer_ids, questions, deadline …)
  artifacts/        NN-<filename> exactly as downloaded
  answers.json      what was submitted; a CORRECT run: the value each field was accepted with (runs)
  answers-first.json  the first pass, when a live fix or an alternative changed it (runs)
  result.json       the GM's reply; `graded`: accepted value per field (runs)
  solver-stamp.json {digest, solver}: the code an autopilot chain ran it with (runs)
  expected.json     graded-correct answers      (fixtures)

a fixture's artifacts/ is a symlink to the cold share (/Volumes/dev_share/documents/huntress/cold/tests/fixtures/..., 2.3 gb only replays read; offload.py move, untracked in git); the record files stay local because the race reads expected.json for the graded vocabulary and kinds.json (the kinds the artifacts carry, written at promote time) for the release-time new-family check, which never opens the artifacts (a tree with no artifacts must not make that check raise, or autopilot never re-runs a fixer's save). a replay, autopilot --check, the rehearsal and the testing grounds need the share mounted (offload.py status: 0 dangling).

8.2 replay

uv run replay loads each fixture's solver, rebuilds the sets with runner.split_sets from the saved artifacts, calls answers_for, and diffs against expected.json. no gm traffic is involved. every tier from day 3 on is a fixture, days 1-2 their singles and timed sets. --benchmark fails any timed fixture whose solve takes over 2s. mutation tests (tests/test_mutations.py) generate variants of every learned rule. tools/comp/rehearse.py (run by autopilot.py --check and per lane in pytest) replays the whole release offline from the same fixtures. tools/after-action/simday.py replays a past day end to end over https, kickoff included, against a git ref plus patches (docs/after-action/after-hours.md "release rehearsal"). simday.py --clean removes every research/simulations/ run, its worktree and the sim/* fix branches once the day's sims are done (after_action stage 7 1a; it refuses while a sim runs). tools/after-action/nightly.py runs the evening's promote / pending / tables / commit / --check gate as one command (6.5).

8.3 table completeness (hlib.tables, fixes pending)

hlib keeps per-day knowledge in tables: decode.DAY_CODECS (line k = day k's codec), apk.CODEC_ALIASES, trace.KNOWN (native tag outcomes), forensics.PROPERTIES, compound.PRODUCERS and the rest. a row a search found at race time is cheap to forget at the end of day, and the cost lands on the next timed tier: a mastery with lines past the table makes _positional give up and the beam search run per set, where the row alone reads every set in well under a second.

two gates make a missed row fail the end of day and the race-morning check:

DAY_CODECS entries may chain steps (morse>hex, decode.day_codec), the same names _new_line's two-step search reports.

8.4 live retest (tools/after-action/testgrounds.py)

simday serves recorded bytes from a fake gm, so it cannot test a change made on the live gm (an organizer fix to a question or an artifact). testgrounds.py replays any exact problem or selected suite from any day against the real gm with a fresh worker that knows only the selected code/doc snapshot. web workers run on the second runner only; other lanes default to the primary runner. the procedure, run only when the human asks, is after_action.md "human-requested: the testing grounds".

a measurement is a control and a treatment (both against the live gm with live variants): the control at the chosen --ref or --as-of snapshot, the treatment with fix commits/overlays under test, compare for a problem or compare-suite for a set.

two topologies (setup --topology):

selftest [--day N --lane L --ref R] runs the race path offline before a live run spends anything: simday's fake gm serves the lane's recorded tiers as the "live" upstream (setup --gm-url), autopilot runs the lane alone (no worker; head's solvers know the day), and it passes only when the single and both tiers flagged, every upstream call of a tier named the one attempt open opened, none was ranked and no call was refused.

9. lifecycles

session start (every agent):

export HUNTRESS_AGENT=<runner>.<role>
python3 tools/comp/preflight.py
python3 coord.py plan
python3 coord.py inbox; python3 coord.py findings

night before: python3 tools/after-action/nightly.py (the day's runs promoted, fixes pending, replay --tables, a committed tree, then autopilot.py --check: every lane's starting solver replays all its fixtures, then the offline release rehearsal; 6.5) and python3 tools/comp/preflight.py (route + clock skew).

release (kickoff, on the gm's clock): first clear stale markers and create a release generation; from t-5m launch a headless fixer for every open lane with none live (6.4). t-65s pin the cloudflare edge (3.6) and measure skew (ntp + date bounds; fall back to local time); t-60s archive the days before (4.2); t-20s warm direct only and start 7 autopilot slots (one per solver lane and a spare; each sends an authenticated list at once and at t-4s); from t-0.3s the 2 hedge connections send the direct list in turn every 80ms and at t+0.05s / +0.15s the 2 fresh ones send one each, first non-empty answer wins; plan in-process, write a generation-matched dayNN.planned (slots claim their lanes); wait <=3s for claims (a single already flagged or carrying autopilot's gaps row counts); write the matching dayNN.live (agents waiting on wait-live --next claim). the phases also go to logs/comp/kickoff-dayNN.log. at t-5m (before the skew sample) the owner checks that the backup kickoff is alive (6.2).

work loop:

id=$(python3 coord.py next)                 # claim (or assist)
uv run newchal --gm-id "$id"                # if no solver yet
uv run dryrun "$id"                          # what yesterday's solver gets wrong
uv run python3 challenges/dayNN/<base>/solve.py "$id"   # chains the timed tiers
python3 coord.py done "$id" <flag>          # already done by the chain

timed tier, end to end:

  1. the solver already passes on the single;
  2. next hands out the time_trial (singles done, or held with the base solved);
  3. the runner takes a shared code-read lease, reruns preflight, warms the route, starts, fetches in parallel, solves, validates and submits within the timer while writers of its solver and its hlib modules are excluded;
  4. on CORRECT: flags.json entry, dashboards refreshed, run saved, and from then on AlreadySolved for anyone else;
  5. the mastery is handed out only after this.

end of day:

  1. reconcile + board; uv run scoreboard --day NN --csv … and uv run timeline --day NN --write (where the seconds went);
  2. promote findings into huntress.md and reusable code into hlib: every reader a solve.py grew during the race moves into its lane's module under the field's exact name, so crosslane (7.5) offers it to every lane;
  3. replay --promote each winning run, then replay (all days must pass); tools/after-action/nightly.py runs the promotion and the gate to the ready stamp in one command after the evening's last commit (6.5);
  4. writeups.

10. failure handling

failure mechanism
vpn / remote vm down direct carries everything; a dead fallback costs one 3s ssh connect, then is skipped for 60s. preflight prints the fix (vpn gui / start remote vm in coursestack) only when direct is down too
timer expires runner retries (fetch or submit), re-fetches the new layout, re-solves
stale questions after retry validate_assignment warns; solvers trust answer_ids / field stems
mastery before time_trial 409 prerequisite_required; next never assigns it
gm rate limit (429) per-team window to the next minute (3.5): 0.4/0.8/1.6s ladder, then every process holds (window_until); starts and probes resend once at the minute; bounded parallel fetch (6), no hedge in the window; shared probe slots
a download 429 the clock cannot outlast AttemptExpired -> the runner retries into a fresh attempt instead of losing the tier on the clock (3.5)
a field another lane already parses hlib.crosslane answers it by exact name, or crashes listing the closest readers (7.5)
the backup kickoff is not running --backup detaches itself; the owner's t-5m check mails both parents (6.2)
a dry run past 3s on a new codec (the dry run's cap is 15s) autopilot.early_handoff: still running after EARLY_HANDOFF_S (3s), autopilot posts that it is searching and releases the claim; it keeps searching and still submits
a third helper on one single coord.TEAM_ASSISTS (2): every runner's assists and pivot assists count together; past it a stall is a report to the human
an organizer fix that turns a field into an enum with rules (a hcfg_storage label) compound._tag_storage(bodies, allowed, rules): the enum's place-naming token, a writable-only clause, base64 / wasm / .hcfg tokens; an enum answer keeps its order and gets the rest of the enum after it; a c++ symbol question that names its form orders the pe candidates
single won by hand probe autopilot follows the single and chains its tiers once solve.py reproduces it; it follows for 22h (a late win still chains)
a field the artifact cannot name (a challenge bug) agents.md "when it fails" 7: derive what and where, guess only variations of a fact read from the source inside the cap, then one observed row and the parent tells the human with a note for the organizer (docs/comp/parents.md ceiling)
a field wrong in several sets of one attempt the bisect sends a found field's namesakes in the other sets alone before halving their groups
agent crashes holding a claim claim stealable after 20 min; reconcile closes flagged ones
two agents solve the same single first CORRECT wins; guard + check stop the other
concurrent writes flock + O_EXCL + atomic replace; tested with 5 simultaneous next calls
runaway log output 64 kib capped, buffered capture in both loggers
the second runner runs out of usage the structured-event guard immediately calls runner model out; repeated handoff sweeps late state
another runner disappears runner <name> low/out manually, or 10 min of silence: claims released with handoff notes, work routed to the other runner
an end of day skips a table row the search that stood in notes it; replay --tables in autopilot --check is not ready, tests/test_tables.py red (8.3)
a live fix wins a timed attempt the run saves the accepted values as answers.json (first pass in answers-first.json), so replay, preflight and the ranked rerun grade against what the gm took (8.1)
429 or blip while warming warm() is best effort and never aborts the run
clock sample unavailable kickoff uses local time; the direct poll remains deadline-bounded and logs each failure
solver crashes on the clock the run is saved with its downloads for offline replay
solver hangs on the clock time_cap: soft cap at the deadline (else 15s), hard kill 5s later; solver_timeout event, parent woken by the stall sweep (docs/comp/timeouts.md)
decompression bomb / huge member hlib.expand budgets end it as a valueerror (a crash row), never a hang or an out-of-memory kill
a shared hlib function is wrong under the clock a fixes.replace in the challenge's own solve.py (no shared lock); a correct publishes it; another lane with the same crash sees FIX AVAILABLE and uv run fixes apply keeps it only if that crash is gone and nothing new broke
an agent keeps working on a task someone else won tools/comp/solved_guard.py refuses its next shell command while the board is open
answer-format wording that traps a substring check answerfmt clause parser, negation first; unsure -> both forms tried on the clock
code changes during timed preflight/run shared code-read leases exclude solver/hlib writers; preflight is uncached and checks the code hash
duplicate/multiple single artifacts indexed files plus single-manifest.json preserve every assignment exactly
dashboard taller than the terminal current autopilot/events render first, newest-first, before variable-height tables
a new label only in timed variants auto alternatives for the new family's label fields, bisected in-attempt
the right value in a shape the grader does not take (guid braces) gm.variants / runner.shape_forms for every answer: a rejected group's field tries its other written forms inside the attempt, the graded one leads preflight, the chain and the fresh bytes (7.6)
the parent dies before spawning its fixers (a classifier stop on turn 1) kickoff's fixer_watch launches a headless fixer per open lane with none live from t-5m (6.4); the parent switches to the fallback model; a restarted parent's first command is spawn --prompts (6.2)
a slot's handout swept into yesterday's archive the archive runs at t-60s and moves only what belongs before the day; the plan's run is a no-op backstop (4.2)
one crashing field in a single-set timed attempt (an empty {} submit otherwise) the crashing fields are isolated by halving and the clean ones submitted; the crashers stay gaps for a fix on the same attempt (7.2)
the next tier started with the code that just crashed chain_waits: a tier waits while its code's digest is that of a crashed run of the challenge, and runs on the next settled save (4.3)
a write in the challenge dir that is no fix re-solves a live attempt autopilot watches solver_files only (4.3); assists' copies go under sessions/patches/work/ (4.5)
junk hex runs swept into a computed-digest field one form per digest; a computed-digest field gets only real member digests (7.6, 7.7)
a parent hand-probing a lane its fixer is working next gives it no assist while the fixer is live and not stalled; gm.hand_cap stops it at one accepted field (4.3, 7.7)
a fixer silent past LIVE_WORK while composing LIVE_HELD (600s) while its edit lock is held or its single is in chain waits (4.1)
a far cloudflare colo on a gm socket (a trace past EDGE_NEAR_MS) the edge pin and the per-dial colo trace re-dial it (3.6)
a handler the end of day did not promote newchal carries it as carried.py; nightly.py stops on fixes pending (7.8, 6.5)

11. known limits

12. tips & tricks

what makes this setup fast to each flag and is not obvious from the gm's api. each line: the trick, why it is there, where it lives.

12.1 the wire

12.2 the release second

12.3 the chain

12.4 the submit

12.5 a single nobody's solver reads

12.6 cpu on the clock

12.7 the agent's shell is the human's shell (verify it, do not assume it)

13. the after-action pipeline

the components table (2), 6.5 and 8.4 describe the after-action tools one at a time; this section is the process that runs them. AFTER_ACTION.md is the procedure: "read after_action.md" hands a fresh reviewing agent the whole end of day, stage after stage, judged by after-hours.md "what we optimize" (a finding counts by the seconds it moved the board finish). nothing runs between 08:15 et and the board complete (variants and rerun refuse it). the first act is the day's checklist, a sed of docs/after-action/checklist-template.md into the report folder: one line per step, ticked - [x] with its evidence (a commit, a record path, n/a: <why>) in the same commit as the work, owner tags splitting the primary runner's items from the web runner's and the human's; docs/after-action/reports/day30/checklist.md is a fully ticked example. long runs go to the background, one commit per item, nothing pushes.

13.1 the stages

stage commands reads produces, and where it lands
1 close the day coord.py reconcile + coord.py board; uv run scoreboard --day NN --csv …/top50.csv, uv run timeline --day NN --write, uv run agent-timeline --day NN --write; the ranked rerun uv run python3 src/ctfkit/rerun.py --day NN; each docs/reference/future-checks.md verify; uv run fixes pending and replay's TABLE … missing row; uv run replay --promote (--promote-single for a probe-won single); python3 tools/after-action/tidy.py sessions/, logs/comp/session.log, flags.json, the public scoreboard, the day's run dirs nothing in progress; top50.csv, timeline.md, agent-timeline.md in the report folder (13.2); ranked runs in each tier's files/runs/ (a non-correct one is a stage 6 item, a new trait the tier's fixture); the future_checks results (13.3); solver-local fixes and table rows moved into hlib with tests; fixtures under tests/fixtures/dayNN/; finished days' runtime files in sessions/archive/day-NN/
2 review uv run afteraction --day NN; bounded grep -n / sed -n over the raw sources the evidence pack, then the records (log lines, result.json, timing.json, intel, the agent timeline, the code path), only then the narratives and commit messages evidence.md, then after-action.md: findings ranked by board-finish seconds, narratives checked, the previous open items re-checked, tomorrow's open items. read-only: no gm start, retry, submit or probe
3 variants -> solvers offload.py status; uv run variants backtest; uv run variants harvest --day NN --keep 0.34 --keep-original (--legacy N for a refilled older day); fix, replay the saved variant, harvest again; backtest again every saved run and harvested variant; the gm's regular pools research/variants/, research/blobs/, research/reports/harvest*.md and backtest.md (13.4); hlib fixes with tests; the day's encodings in tests/test_readiness.py; a regraded family in runner.REGRADED
4 tomorrow's hints uv run variants prompts; the upcoming lines of python3 tools/comp/autopilot.py --check (6.5) every start.json prompt and question; the hlib registries research/reports/prompts.md; prep in hlib with tests; ungraded fields in docs/comp/lanes/<lane>-prep.md; tomorrow's docs/reference/future-checks.md (13.3)
5 inefficiencies the timeline's phases and the agent timeline's checks; uv run python3 tools/after-action/profile_solvers.py --days NN; uv run python3 tools/after-action/simday.py --day NN --ref HEAD --agents 5 --parents (+ --break <lane>) and simtimeline.py <run>; the live-agent replay (simday --llm … --spawn at the morning ref, then with the fix commits); autopilot --check and pytest wall times the timelines, every non-web fixture, the tree at a ref docs/after-action/reports/cross-day/solver-profile-<date>.md; runs in research/simulations/ (13.5); every race-path cost a stage 6 item with its seconds
6 fix per item: a test that fails without the fix where practical, uv run pytest -q, the touched hlib module's release replay, autopilot --check ready for race machinery, a sim or timeline number re-run the report's findings and open items, board-finish order, race logic before solver speed one commit per item with its docs (capabilities for the mechanism, agents / huntress / briefs for what agents do); the open item fixed <commit>, the finding fixed in <commit>; a bigger or riskier item becomes a decision
7 gate, tell the human 13.7 the committed tree the ready stamp for tomorrow, the disk freed, a docs: commit, the short report to the human; a push due
8 the blog a writeup.md per challenge; the blog guide's posts; blog/site/format_prose.py, blog/site/check_answers.py (+ --full), blog/site/publish.py the report, the challenge dirs challenges/dayNN/<id>/writeup.md; the day's posts in blog-full/ then blog/; a ## day NN additions section on every changed tool page; published only when the human asks

the web lane's half of stages 1-8 (its fixtures, review, harvest, prep, fixes, posts) is the web runner's, on its own checklist lines. the testing grounds (13.6) is no stage: it runs only when the human asks.

13.2 the report folder

path written by holds
docs/after-action/reports/dayNN/after-action.md the reviewer, stage 2 on, kept current to stage 7 the ## end of day log (stage, status, produced), the summary (finish, place, critical lane), findings F1.. (impact in seconds and how computed, evidence path:line, mechanism, fix, confidence, seen before; fixed in <commit>), narrative claims checked, decisions D1.. (the human's calls, each with a recommendation), previous open items, and ## open items: one ddNN-<n> row each with owner area, impact (s) and status (open, fixed <commit>, decision, wontfix <why>, obsolete <why>); tomorrow's evidence pack machine-reads that table, so its columns keep their order
…/dayNN/checklist.md the template's sed, ticked per step 13.1's stages as one line each with its evidence
…/dayNN/evidence.md uv run afteraction --day NN records, never conclusions: the timeline's result, lanes and where to invest, the runbook checks, every tier's attempts, claims and handovers, gm incidents, stall alerts, urgent mail, the value-in-hand latency, code changed since the last review, the previous open items, the raw-source index and the narratives list
…/dayNN/timeline.md uv run timeline --day NN --write every tier cut into phases that add up exactly to prerequisite flag -> flag (the rest printed as residual), the critical lane, what each cause cost the finish, ## fixer tasks (every non-web tier handed to an llm agent is an open item)
…/dayNN/agent-timeline.md uv run agent-timeline --day NN --write per challenge section, what each agent did in order, and the checks where an agent did other than the runbook asks
…/dayNN/top50.csv uv run scoreboard --day NN --csv the public board's top-50 solves of the day
…/day30/prompt-coverage.md hand-written audit every non-web prompt and question set of days 01-30 against what a later day graded, and the solver path for each hint never graded; the machine half is uv run variants prompts -> research/reports/prompts.md
docs/after-action/reports/cross-day/ profile_solvers.py, cross-day reviews solver-profile-<date>.md (one a night), patterns.md, the 429 analysis, the rival boards
docs/after-action/reports/bug_reports/ the reviewer reports sent to the organizer
research/reports/backtest.md uv run variants backtest per lane ok / wrong / crash / unchecked over every saved run and variant (13.4)

13.3 docs/reference/future-checks.md

13.4 variants: harvest, backtest, prompts

after-hours.md "variant research" records how the gm hands out variants; the 6.3 row lists the subcommands.

13.5 simulations and drills

drill spec budget
clean '' 15s
each lane broken (five drills) <lane> 50s: 15 + two fixer turns of --think + 15 for autopilot's re-run
one mastery break <lane>:mastery, the lane rotating by day (day % 5) 50s
rate limit rate-limit:57 (a first 429 after 57 calls in the minute) 90s: 15 + the 60s window + 15

13.6 testing grounds (the live retest)

the mechanism is 8.4, the procedure after_action.md "human-requested: the testing grounds". no stage: only when the human asks, never in a race. auto mode refuses the live steps (gm submits, spawning the worker), so the human runs open and launch*.

tools/after-action/testgrounds.py what
setup / setup-suite a control at --ref / --as-of (default: the last commit before that day's 09:00 et), or a treatment (--control RUN --patch SHA/A..B/all, --overlay); --topology race/worker; a suite is one isolated run per base
leakcheck RUN --answers the graded answers of the fields under test searched in every file of the tree; open refuses without it
open live: one fresh attempt per timed tier
launch-race / launch race topology (the split gm testgrounds_gm.py, kickoff, autopilot, the fixer) / worker topology
frozen / frozen-web offline a/b in one process: a cold worker against the challenge's recorded fixture served by simday, or live local web targets behind an offline simday gm
selftest the race path offline, no worker, before the first live run after a harness change
note, report, check [--draft], published a mid-run harness fix; the record; record complete; the artifact link
compare / compare-suite compare.md: each metric's median and range per arm and the delta
clean RUN… / --all a complete record's tree goes, report.html's sha256 kept in meta.json

13.7 the gate

stage 7, in the checklist's order (its items 7.1-7.9, 7.2b included):

  1. uv run pytest -q green.
  2. uv run variants backtest: 0 wrong, 0 crash (13.4).
  3. python3 tools/after-action/nightly.py (6.5): (1) promote the day's correct non-web runs with no fixture, (2) uv run fixes pending empty, (3) uv run replay --tables with no row a search supplied (8.3), (4) nothing uncommitted under the checked paths, (5) autopilot.py --check ready and stamped; it stops at the first failure with what to do.
  4. the ready stamp, sessions/ready.json, is keyed on the committed tree of readystamp.CHECKED_PATHS (src/ctfkit/readystamp.py): src, tools, tests, challenges, gm.py, coord.py, AGENTS.md, START.md, WEB.md, HUNTRESS.md, the second runner's session doc under docs/comp/, docs/comp/lanes/web.md, docs/comp/lanes/web-prep.md, pyproject.toml, uv.lock. a commit under any of them voids it, so nightly runs again after the evening's last commit by anyone (a web runner's late commit included), or race morning pays the check.
  5. one clean simday at head, board finish no worse than the day's earlier sim.
  6. drillgate.py 8/8 (13.5).
  7. the disk: python3 tools/after-action/offload.py move (13.8) and uv cache prune; offload.py status ends with the free space (the prune waits while a uv run dash holds the uv lock, never --force).
  8. sims and trees: simday.py --clean, testgrounds.py clean --all, once every number the report cites is in it.
  9. docs current: for every commit since the morning, capabilities.md, start.md, huntress.md, agents.md, docs/comp/parents.md and each lane's brief and prep file describe it as it now works; a docs: commit, then autopilot --check again if head moved.
  10. tell the human, short: finish and place, top findings and fixes (commits, measured effect), the decisions with a recommendation each, what is open, and that a push is due (commits since origin/main): the push is the human's, nothing in the pipeline pushes.

the board-gated steps (the ranked rerun, harvest, tidy, simday, drillgate, offload, the stamp) wait for the web lane: a non-web after-action that starts while web still races works on a ddNN-fixes sibling worktree and runs them after the merge (after_action.md "always").

13.8 the research/ layout

path in git what kept
research/simulations/<stamp>-dayNN/ no simday runs (13.5) removed by simday.py --clean the same evening
research/testing_grounds/<id>/ the record; tree/ and harness/ no live retests (13.6), with suites/ and templates/ the record for good; the tree until clean
research/reports/ yes only what uv run variants writes: backtest.md, harvest.md + harvest-dayNN.md, prompts.md each run rewrites its file
research/variants/dayNN/ no harvested variants as run dirs past days on the share
research/blobs/ no harvested artifacts once by sha (the harvest hard-links into it) local, never offloaded
research/experiments/ yes opt-in experiments (variant probes, the pattern lab) run only when the human asks

the share (tools/after-action/offload.py status|plan|move|localize [--keep N], the components table) is an smb volume mounted by hand. move copies the cold data, sha256-verified, and leaves a symlink: past days' challenges/dayNN/*/files (solve.py stays), research/variants/dayNN, an archived day's session.log and scoreboard/, every fixture's artifacts (the race reads only expected.json). never offloaded: the newest --keep days, live sessions/, archived intel, mail and markers, research/blobs/, the uv cache. unmounted, the links dangle and the backtest skips those days: stage 3 starts with offload.py status.

{generated by a clanker}