tooling: replay fixtures and regression tests
every lane changed one thing a day while keeping everything before it. these tools keep yesterday's answers correct while today's change is added.
replay fixtures
every correct timed run is saved with its assignment, every downloaded artifact, the submitted answers and the grader's result. a promoted run becomes a fixture: replay re-splits the deliveries, re-runs the parser and compares every answer with the graded one. singles won by per-field probing become fixtures from the saved assignment plus the confirmed values. a benchmark mode times split-and-solve per fixture and fails any timed fixture over a two-second budget. as of day 4: 41 fixtures across all tiers of days 1-4.
labels and triage
labels tables every graded value of a field against features of its artifact, which is how rules like "expansion follows the fragment family" were found instead of argued. triage emits a json report per artifact (format, elf/pe/ .net tables, apk/dex, text properties, embedded configs) and a token list for label guessing.
mutation tests
fixtures pin exact artifacts; mutation tests pin the learned rules on generated variants: decoy descriptors, case, layouts, every encoding, every representation and transform pair, decoy control blocks, keyed codecs, lying headers, unknown day tags. 246 tests as of day 4.
day 16 additions
the table check
several parsers keep a table that gains a row a day, with a search behind it for the row that is not there yet: the decoder's line schedule, the apk toolkit's codec names and the binary toolkit's known dynamic-trace tags. the search makes a missing row invisible. the run passes, and nothing asks for the row until the day the search can no longer stand in for it. day 15's rot47 line went that way, and day 16's warmups mastery paid for it (dot com).
each of those searches now records the row it stood in for:
@dataclass(frozen=True)
class Row:
table: str
key: str
value: str
def note(table, key, value):
ROWS[Row(table, key, value)] = None
replay --tables takes the record after every fixture and fails the fixture when it holds a row, printing the row in the form it should be added (decode.DAY_CODECS[15] = 'rot47'). the end-of-day regression run replays every category's fixtures this way. run against the table of the morning of day 16, the day 15 mastery fixture fails with exactly that row.
the second half catches the opposite case: a row that exists only in one day's solver. the check imports the solver, compares every module-level dict, list, tuple and set of the shared library before and after the import, and lists what changed, plus any per-field handlers the solver registers. there is no list of tables to keep current; a new table is covered the day it appears. a test stays red until every such row is in the library.
before = tables() # every module-level container of the library
load(solver_path)
after = tables()
rows = [row for name in after if name in before and before[name] != after[name]
for row in diff(name, before[name], after[name])]
day 16 promoted
every tier of day 16 is a fixture. the warmups mastery run's saved answers were rebuilt from the values the grader accepted, because the run's own file held the first, rejected answers (see the solver pipeline).
day 22 additions
a rejected run says what changed
a run the grader rejected has no complete answer key, so its replay used to print only that it was unchecked. the saved run does hold the grader's per-field verdicts: the answers it accepted and, for a field rejected in every form, the forms that were tried. replay now compares against those: how many accepted answers the current parser still produces (and which moved), and for each rejected field whether it now produces a value not yet tried. that is the question a parser fix has to answer before the next attempt.
day 23 additions
a saved web run belongs to its release day
some web challenges reuse a base whose identifier includes an earlier curriculum day. selecting regression runs by that identifier omitted runs created on day 23. the fixture index now uses the recorded release day when it exists and falls back to the identifier for older archived runs.
release_day = completed_release_day.get(base_id) or day_from_id(base_id)
if release_day == requested_day:
replay(saved_run)
this keeps an offline replay set tied to when the specimen was received, including its timed variants, without changing the saved evidence.
day 24 additions
a passing regression run, keyed on the code it checked
the full readiness check, a replay of saved and modelled specimens through every solver, takes about three minutes. a pass is recorded with a key, and a later check whose key matches is instant. the key used to be the repository's head commit, so a commit that touched only notes or reports invalidated a pass that still held. the key is now the committed tree of the paths the check actually exercises (sources, tools, tests, solvers, lockfiles), one git tree id per path; a commit elsewhere keeps the pass, and uncommitted edits under a checked path always force the full run.
CHECKED_PATHS = ("src", "tools", "tests", "challenges", ..., "pyproject.toml", "uv.lock")
def checked_tree():
return git("rev-parse", *[f"HEAD:{p}" for p in CHECKED_PATHS]).replace("\n", "+")
def stale(day):
row = read_stamp()
if row is None or row["day"] != day:
return "no pass for this day"
if row["tree"] != checked_tree():
return "the pass predates the last commit under a checked path"
if git("status", "--porcelain", "--", *CHECKED_PATHS):
return "uncommitted edits under the checked paths"
return ""
the next-day model learns the new row
the readiness tests build the blobs the next warmups day could bring, one encoder per row of the decoder's schedule. day 24 added the netstring row (decoding engine), so the model gained a netstring encoder for line 24; with 24 schedule rows and only 23 encoders, the modelled blobs no longer matched the schedule and 21 cases failed.
day 28 additions
the end-of-day regression gate as one command
the end of each day runs the same checks before the next morning: every correct timed run of the day becomes a replay fixture, rows a day's solver still carries outside the shared library are listed, the day's fixtures replay through the shared readers with the table check, and the readiness check runs for the next day and records its pass, keyed on the committed tree as on day 24. on day 28 that sequence, done by hand, failed twice in a way the order explains: an uncommitted file under a checked path withheld the pass, and a freshly promoted fixture solved through the search because its schedule row was not added yet. one command now runs the steps in order and stops at the first failing one with a single line of what to do:
STEPS = (
("promote", promote_correct_runs), # the earliest correct run per mode
("pending", pending_rows), # rows still outside the library
("tables", replay_tables), # a search standing in for a row fails
("committed", uncommitted_checked_paths), # "commit first"
("readiness", readiness_for_next_day), # the pass is recorded and read back
)
for name, step in STEPS:
problem = step(day)
if problem:
sys.exit(f"{name}: {problem}")
the next-day model learns row 28
day 28 added reversed hex as the decoder's row 28 (decoding engine), so the next-day blob model gained a reversed-hex encoder for line 28; without it the modelled blobs no longer matched the schedule and every warmups readiness case built on them failed.
day 30 additions
families named in advance
both day-30 losses were families the organizer named in the prompt or the question text that no reader read: a configuration record under a renamed magic (binary analysis) and the relation family (archive and metadata readers). the readiness check's look-ahead read only the values answer enums offer. it now also reads every non-web day's questions and prints a line for a record magic a question names that no library source names, a field family an answer id names that no reader covers, and, on each lane's newest day, a member kind the questions name whose members sniff as plain binary and the kinds the prompt lists. a phrase counts as known when all its words share one source line. against the tree of the morning of day 30 it names the magic, the relation family and the prompt's four kinds; at the end of day 30, nothing. it takes about three seconds and never blocks the check.
a written audit of all 145 non-web prompt and question files of days 1 to 30 (prompt-coverage.md) lists what each named, whether a later day asked for it, and the solver path for every hint never asked: metadata 26 covered, 8 partly, 0 not; script 2, 3 and 19; binary 5, 4 and 4. it ends in a watch list for day 31, and the next day's checks were written beside it, each with a command that verifies it.
the forward matrix
the tests now build the cases a next day could bring rather than the ones seen: a configuration record under a new magic in every carrier (raw, base64, hex, utf-16, xor, compressed, overlay, a jar's tail) and member kind, every cell read with no false read, and every relation kind over the day's corpora. the day's full regression replayed every fixture (448 of 448) and the per-lane variant backtest found no wrong answer and no crash in any lane.
the next-day model learns row 30
day 30 added url twice as the decoder's row 30 (decoding engine), so the next-day blob model gained a percent-twice encoder for line 30; with the model one day short, its synthetic day-31 line sat on the slot the table now reads as url twice, and four warmups readiness cases failed.
a base with no timed tier is no lane base
the end-of-day gate promotes the day's correct runs (day 28). on day 30 it promoted the organizer's one-off static challenge (one flag, no timed tiers, a lane prefix in its id) as a binary fixture; the lane solver cannot read it, so every library release replay and the readiness check failed on it. a base that owns no timed tier task is no longer a lane base (with no task store, every base still counts), and the fixture is gone. the fifteen day-30 lane fixtures were promoted, and the gate passed.
changelog
- day 2: saved runs and offline replay of a solver against a run.
- day 3: fixtures and promotion, the benchmark gate,
labels,triage, the first mutation tests. - day 4: every tier of day 4 promoted, mutation tests for every new rule.
- day 5: every tier of day 5 promoted; mutation tests for the new decoding, png, rust and multi-word field rules.
- day 7: a codec grid test (every representation, layout, reversal and transform the descriptor finder must read: 1008 specimens), compiled-carrier samples, header-field cold solves of earlier days, and end-to-end bundles in languages not yet seen.
- day 9: every tier of day 9 promoted; a backtest of each lane's newest parser over every saved run; readiness tests for each lane's likely next change and its break (an encoding, a header field, a runtime, a carrier); a test suite that cannot reach the network.
- day 16: a table check: a search records the table row it stood in for and the regression replay fails on it; rows a solver adds to the shared library are listed until they move into it; every tier of day 16 promoted.
- day 22: a rejected run's replay compares against the grader's saved verdicts: accepted answers still produced, rejected fields' new values.
- day 23: web regression selection uses the recorded release day, with the identifier day as a fallback for older saved runs.
- day 24: a passing readiness check keyed on the committed tree of the paths it exercises; a netstring encoder in the next-day blob model.
- day 28: the end-of-day regression gate as one command (promote the day's correct runs, list rows outside the library, replay the day's fixtures with the table check, refuse uncommitted checked paths, run and record the readiness check); a reversed-hex encoder in the next-day blob model.
- day 30: the readiness check names record magics, field families, member kinds and prompt kinds no reader covers; a written audit of every non-web prompt; a forward matrix of record magics by carrier and member kind and of relation kinds; a percent-twice encoder in the next-day blob model; a base with no timed tier is no lane base.
limitations
fixtures only pin variants we have seen; mutation tests generalize the learned rules but cannot predict a new family.
{generated by a clanker}