tooling: elf, c++, hcfg, pe/cli, and cumulative binary dispatch

the binary lane accumulated formats across days. our toolkit combines small binary-structure readers with gnu binutils where textual disassembly is more practical. this article documents the custom logic referenced by the challenge writeups.

elf orientation

the elf reader gets class and byte order from e_ident[EI_CLASS] and e_ident[EI_DATA], then unpacks e_type, e_machine, and e_entry using the declared width and endianness. it maps machine and type numbers to the exact grader vocabulary ([redacted]).

gnu readelf supplies section and symbol views. a designated exported symbol is chosen from defined global FUNC symbols after filtering runtime names. the .huntress marker comes from that section's string content. gnu tools are selected explicitly because llvm's macos output differs enough to break text parsers.

to recover a simple constant-return function, the tool scopes disassembly to the named function and recognizes a move-immediate into eax/rax, converting the immediate to a decimal answer.

raw section and symbol tables

when table order itself became graded, tool output was no longer enough. the parser reads elf32 or elf64 section headers in either byte order:

  1. read e_shoff, e_shentsize, e_shnum, and e_shstrndx.
  2. parse each section header using the class-specific structure.
  3. follow the section-name table and resolve every sh_name.
  4. find the SHT_SYMTAB section.
  5. follow its linked string table and resolve each entry's st_name in table order, then apply the filtering rules the grader expects ([redacted]).

disassembly uses intel syntax without raw instruction bytes. standalone hex immediates are normalized to decimal before the instruction list is serialized as compact json.

c++ abi and coff symbols

for elf c++, symbol names beginning _ZTV identify vtables and _ZTI type information under the itanium abi. standard-library and __cxxabiv1 entries are excluded. method recognition uses the encoded method length and name; for example, a const score method matches _ZNK...5scoreE....

mingw pe also uses itanium-style names in its coff symbol table; the pe analyzer emits them in the specific form that was graded ([redacted]).

hcfg: a validated embedded configuration

the native and managed samples shared this byte layout:

offset  size  meaning
0       4     ASCII "HCFG"
4       1     version
5       1     single-byte XOR key
6       2     ciphertext length, little-endian
8       4     CRC32 of plaintext, little-endian
12      N     XOR-encoded JSON

the parser tests every occurrence of the magic. for each candidate it verifies bounds, xors exactly N bytes, checks crc32 over plaintext, parses json, and requires a json object. if no candidate validates it fails; if multiple different valid configurations exist it reports ambiguity. this layered validation makes a whole-file scan safe even when the magic also appears in a string or symbol name.

length, expected_crc = unpack_from("<HI", blob, offset + 6)
plain = bytes(byte ^ key for byte in blob[offset + 12:offset + 12 + length])
assert crc32(plain) == expected_crc
config = json.loads(plain)
assert isinstance(config, dict)

pe and cli metadata

a managed assembly begins as a pe. the parser follows the dos header's e_lfanew, verifies PE\0\0, reads the coff machine, and locates data-directory entry 14—the com descriptor. a nonzero rva indicates a cli image.

rva-to-file-offset conversion walks section virtual addresses and raw pointers. the cor20 header points to a metadata root that must begin BSJB. from there we read the runtime version, metadata-version string, and aligned stream headers for heaps such as #~, #Strings, #US, #GUID, and #Blob. this is enough to label the file [redacted], report [redacted], and recover [redacted] without loading the clr.

our fuller ecma-335 reader parses row counts and variable-width indices in the #~ stream. it resolves type, method, field, fieldrva, and manifest-resource tables. static field initializers are useful for finding an hcfg byte array; the validated whole-file hcfg scan remains a safe fallback.

cumulative dispatch

cumulative archives prefix members d01-, d02-, and so on, while answer fields begin d01_, d02_, and so on. the dispatcher groups non-archive members by tag while preserving archive order. it computes artifact_sha256_K from the kth member and sends analysis fields to a registry:

ANALYZERS = {
    "01": analyze_elf,
    "02": analyze_cpp_hcfg,
    "03": analyze_dotnet,
}

adding a cumulative day means registering one analyzer, not modifying every previous branch. unknown tags fail loudly. this registry is the “final solver grouping” referenced in table manners: it is a small dispatch architecture, not a hidden decompiler or external product.

day 4 additions

field producers instead of day analyzers

the day-4 tag was a go build, elf on the single and both elf and pe in mastery. every one of its field names (format, architecture, config fields, crc) already existed under earlier tags, yet the solver failed until a d04 analyzer was registered. on day 4 the per-day analyzers then became one registry keyed by field name, each producer taking the tag's main binary:

PRODUCERS = {
    "format": format_of,               # elf, pe, [redacted]
    "machine": machine_of,             # elf machine or managed coff machine
    "architecture": architecture,      # [redacted] for elf and pe alike
    "metadata_version": cli_version,
    "section_names": section_names,
    "symbols": symtab_names,
    "disassembly": exported_disassembly,
}

config fields come from the validated hcfg decoder and elf triage fields from the header reader. a tag with no registered analyzer uses the registry; an unknown field name raises an error that names it. the same registry replays every binary fixture from days 1-4. architecture reads the elf e_machine or the pe coff machine and maps both to the platform name ([redacted]), not go's amd64.

build facts

the producer registry also answers the build facts a binary question tends to ask, each read from structure rather than tool text:

on the day-4 bundle they report the go build as go, static, stripped, with its toolchain version, and the managed assembly as dotnet.

day 5 additions

day 5 added rust-specific analysis:

day 6 additions

day 6's member was a java archive. two things changed:

a malformed or truncated class is not fatal: its pool raises a bounded error naming the offset, the class is scanned as raw bytes instead, and the other classes still parse. the file-type sniffer names zip-based formats by their marker files (java archives, python wheels, nuget packages, epub, opendocument, editor and browser extensions) and reports any other zip member as a zip, so an archive-like member is identified instead of skipped.

day 7 additions

naming the format from the bytes

the format producer returned the elf label for anything that was neither a jar nor a pe, and an apk passed the jar check because it carries META-INF/MANIFEST.MF. a new runtime would have been answered with a well-formed wrong label. formats are now named from their bytes (apk, dex, macho, macho-fat, [redacted]) with a warning that the label is unseen, and those members are accepted as a day's main file instead of being rejected.

dex constants

an hcfg blob kept as a base64 constant in a dex file is found through the parsed string table; scanning the raw string data fails when the uleb128 length byte is printable and joins the base64 run.

mach-o

a stdlib reader walks the load commands of a thin image or the first slice of a universal binary: cpu type (in the same vocabulary as the elf answers, [redacted]), sections in load order, the symbol table, and disassembly of the function the question names, through gnu objdump. mach-o symbols carry no size, so the listing is cut after the last ret before alignment padding. checked against clang-built intel and arm binaries, otool -l, nm and lipo.

webassembly

a stdlib reader lists sections (custom ones by name), imports as module.name, exports, and function names from the name section. no disassembler is installed, so a disassembly field raises instead of guessing.

failures are errors

a binutils call that fails used to return its empty stdout; a disassembly or symbol list then came back empty, which is a valid-looking wrong answer. tool failures now raise with the tool's message (an empty symbol table from nm on a stripped file is the one empty result that is real).

day 8 additions

hcfg in a base64 literal

a swift elf kept its hcfg blob as a base64 string literal, and the bytes HCFG also appeared in its code as an instruction operand. the validated decoder rejects the operand (its length, checksum and json do not check out), and the literal is found by its encoded prefix: base64 of anything that starts with HCFG starts with SENGR, so the search decodes only literals that begin there instead of every base64-looking run in a 67 mb file (1.2 million candidates). the decoded configuration is cached per binary, since every config_* field asks for it.

answers by kind

the same field name can be asked as a label on one day and as an integer on another. producers are chosen by the question's answer kind ([redacted]). ids with a set prefix (set3.d08_…) are normalized before the producer lookup, so mastery sets take the same path as the single.

fields for old tags

has_cpp_vtable and has_cpp_typeinfo come from the c++ abi symbols (_ZTV, _ZTI, standard-library ones excluded) and are false for a jar. compression_methods comes from the zip method ids over a jar's entries ([redacted]). pe members also answer section_names.

faster on large members

header-only elf fields (class, endianness, machine, type, entry point) are read from the header bytes without starting binutils, the remaining tool runs share one temporary copy of the binary, the sha-256 is computed without md5 and sha-1 beside it, and a set's day tags are analyzed in parallel. the day 8 mastery fixture went from 3.4s to 1.7s.

day 9 additions

the configuration blob had arrived raw, as a .net byte array, as java and dex string constants, and as base64 literals in swift and nim builds. day 9 replaced the per-format lookups with one search over every member of a tag, the main binary first; a zip-based member (jar, apk, nested zip) is searched entry by entry. forms run from cheapest to most expensive, and a later form runs only when no earlier one produced a valid blob:

  1. raw HCFG occurrences.
  2. base64 anchored on the encoded magic (SENGR), standard or url-safe, padded or not. go packs string data back to back, so an unpadded literal can run into the next string; its whole 4-character groups are decoded too.
  3. hex of the magic, either case.
  4. both text forms as utf-16le, the shape of a .net #US string or a windows wide literal.
  5. the whole blob xor'd with one byte. HCFG ^ k keeps the neighbour differences 0b 05 01 for every key, so one pass over the member finds every key at once instead of 255 searches.
  6. zlib, gzip, bz2 and xz streams and deflated zip entries inside the member (an appended archive, a packed resource), inflated in 64 kb chunks under a 64 mb cap, so a false header in a large binary fails on its first chunk.
  7. every long base64 literal, the blob anywhere inside it: the slow last resort.
def _xor(d: bytes) -> list[Hit]:
    n = int.from_bytes(d, "big")
    diff = (n ^ (n >> 8)).to_bytes(len(d), "big")   # diff[i] = d[i-1] ^ d[i]
    at = diff.find(b"\x0b\x05\x01")                 # H^C, C^F, F^G, any key
    ...                                             # key = d[at-1] ^ ord("H")

every candidate goes through the same validator (bounds, crc32 over the plaintext, a json object), and two different valid blobs in one tag raise instead of one being chosen.

the offset

hcfg_offset reports where the carrier sits ([redacted]). readings that could also be meant, inside or outside a container, before or after a prefix, are kept as ranked alternatives behind the first one, and a jar's carrier is read inside its class file like any other member.

old tags answer from their format or fail

each day adds field names, and older tags can be asked the new ones. asking every producer for every field of every tag in the day 9 bundle found well-formed wrong answers: a jar's architecture and machine came back as 0x0 (zip bytes read as an elf header), function counts and main-function lists were zero and empty for c, c++, nim, swift and jar builds, the swift build was read as rust for crate name and mangling, the autoit version was empty, and a jar's language was c. each now answers from the member's own format or raises with the member kind. a tag whose member no reader handles still answers its configuration fields, since the carrier search ignores format.

faster across sets

the day 9 mastery fixture's solve went from 1.68s to 0.70s.

stripped nim builds

nim is recognized by its entry symbols (NimMain), which strip removes with the symbol table. a stripped build still carries nim's runtime strings in .rodata: fatal.nim and the other standard library source names in fatal errors, the segfault and unhandled-exception messages, the defect type names, and an assertion's <file>.nim(line, col) location. the language check reads nim when two of these independent groups agree, after the runtime symbols and before the c++ test; one nim word in a c or c++ build is not enough.

day 10 additions

section-table questions

day 10 asked d10_analysis_has_hcfg_section. any field shaped has_<x>_section now asks whether a section named x exists, over every section table the member has: elf and pe section names, each slice of a mach-o, and webassembly sections. a leading . or __ is removed, case is folded, and dotted names compare in snake case (.gnu.hash is gnu_hash); whole names only. a member with no section table (a jar, a dex, an elf without section headers) raises instead of answering false.

runtime markers that survive strip

a stripped static zig build keeps none of the symbols our zig markers looked for, and both day-10 builds read as c. zig is now recognized by its standard library's panic and debug strings in read-only data, two of which must be present, because c builds compiled with clang's sanitizers carry similar strings. the c++ markers were checked against real stripped builds the same day: libc's own c++ teardown hooks (the __cxa_ finalize and atexit entry points) no longer read as c++, since every c program links them; a stripped static c++ build, a mingw c++ pe (by the itanium abi's typeinfo names) and an msvc c++ build read as c++, and an msvc c build stays c.

jvm, kotlin and swift structure

for questions about a jar's or a swift build's own structure, three readers were added:

the dynamic trace

every tag since day 2 prints one json line describing a modeled run, and an environment variable chooses which outcome the program models. the trace reader recovers the line's schema, keys and print template from the bytes in every build style (a printf format, rust's format pieces, a kotlin string concatenation, swift and nim string pieces, a .net serializer call read from its il), normalized to one template form. the outcome and event count for each environment value come from the code itself for jar and .net tags: a small interpreter runs the method that compares the variable's value and returns the branch. native tags use outcomes recorded from runs of the known builds, only while the tag's signature strings are in the bytes; an unseen native tag raises instead of guessing.

day 11 additions

where an offset sits

day 9 asked for the file offset of the configuration blob and day 10 whether it has a section of its own. hlib.placement answers the next questions about the same bytes: which section holds a file offset, and the virtual address it loads at, for elf, pe and thin mach-o files.

def region_at(b: bytes, offset: int) -> Region:
    hits = [r for r in regions(b) if r.offset <= offset < r.offset + r.size]
    if len(hits) != 1:
        raise ValueError(f"file offset {offset} is in sections {[r.name for r in hits]}")
    return hits[0]


def address_at(b: bytes, offset: int) -> int:
    region = region_at(b, offset)
    return region.address + offset - region.offset

each region is a section's file range and its address; sections with no bytes in the file (.bss) are left out, and an offset in no section or in two raises. the producers read the carrier's first byte and answer its section name, its virtual address, and, for a pe only, its rva.

fewer external tool runs

the c++ marker fields ran nm (and objdump -t on a pe) once per field, and the disassembly field ran objdump for every set even when the sets shared a binary. symbol tables and disassembly are now cached per binary, which cut the external processes of the heaviest mastery fixture from 28 to 12. a jar's zip index is parsed once for every reader that opens it.

day 12 additions

fields from the other lanes

the generic dispatcher answers a field from the tag's own format, the hcfg blob or the dynamic trace. a name none of them produces now goes to cross-lane field resolution before the set fails, so a metadata or apk property asked about a binary reuses the reader that lane already has:

for n in sorted(names - out.keys()):
    try:
        out[n] = crosslane.resolve(n, b, "", skip="binary")
    except KeyError as e:
        misses.append(str(e.args[0]))

the error for a field nobody answers names the tag's main member and the closest readers each lane offered.

.net columns by name

the ecma-335 table reader (day 3) returned rows as tuples, so every .net field was a hand-written index. dotnet.named reads the one-row Module and Assembly tables by their ecma-335 column names, following each column's heap: strings from #Strings, guids from #GUID (1-based, zero means empty), blobs from #Blob as hex:

for column, kind, v in zip(columns, _SCHEMA[tid], rows[0], strict=True):
    key = f"{table}-{column}"
    if kind == _S:
        out[key] = _str(t, v)
    elif kind == _G:
        out[key] = _guid(t, v)
    elif kind == _B:
        out[key] = _blob(t, v).hex()
    else:
        out[key] = str(v)

the same map carries the joined assembly-version, the assembly's string custom attributes (target framework, file and informational versions, configuration, company, product, title; each value a length-prefixed string read from the attribute blob) and the names of the referenced assemblies. triage prints the map as columns for any managed pe.

day 15 additions

two changes, both about which member of a bundle answers a field.

a role in the field name. a day tag may hold two builds of one program (dNN--variant-NNN-stripped.elf and -symbolized.elf), and a field dNN_analysis_<role>_<field> then asks <field> of one of them. the role is a word that names exactly one member of the tag, a member with a sibling whose file name differs by that word alone; the field is answered from that member as an ordinary <field>. two guards keep ordinary names safe: a name a producer already answers is never split (main_class stays one field even when the tag holds Main.class and Other.class), and the part after the role must itself be a name a producer answers.

def role(files, tag, name):
    head, _, rest = name.partition("_")
    if not rest or named(name) or not named(rest):
        return None
    tagged = {rel: b for rel, b in files.items() if tag_of(rel) == tag}
    found = roles(head, tagged)   # one member named by the word, with a sibling
    return (rest, tagged[found[0].member]) if found else None

functions of a c build. function_count was read for go (pclntab) and rust only. a c elf now ranks three readings, each a fallback for the one before: every STT_FUNC symbol when .symtab is there, the program's own (the c runtime's helpers removed), then the starts listed in .eh_frame_hdr's binary search table, which stripping keeps. without an unwind table the symbol readings still answer; with neither, the count fails and says which is missing. the table is read for the encodings gcc and clang write (a udata4 count, datarel | sdata4 entries relative to the section); any other encoding raises. the count is answered as a low-confidence candidate list, never as a confident value.

version, ptr_enc, count_enc, table_enc = hdr[0:4]   # 1, *, 0x03, 0x3b
count = u32(hdr, 4 + ptr_size)
starts = [hdr_vaddr + s32(hdr, 8 + ptr_size + 8 * i) for i in range(count)]

day 16 additions

day 16's tag was two webassembly builds of one function, answered by the day 15 role routing with no new code (wasm up). its prompt asked to "map every door in this module": imports, memory and indirect calls, none of them asked as a field yet. the webassembly reader (day 7: sections, imports, exports, names) now reads every door.

sections. imports with their kind and, for an imported table or memory, its limits; exports with kind and index; the table and memory index spaces (imported first, limits in slots and 64 kib pages, the maximum absent when unbounded); the start function; element segments in all eight flag forms (active, passive, declarative; function indices or ref.func expressions, ref.null slots left out); data segments with mode, memory and constant offset; and the name section's function index map.

instructions, decoded. call and indirect-call counts come from walking each function body instruction by instruction, every immediate skipped by its encoding, never from scanning bytes. i32.const 17 is the bytes 41 11, and 0x11 is also the opcode of call_indirect; a byte scan of day 16's modules would have reported an indirect call that does not exist. the decoder knows the mvp set plus sign extension, multi-value, reference types, bulk memory, tail calls, exceptions, simd, threads and multi-memory, and raises on anything newer instead of miscounting.

def opcodes(mod):
    body, count, q = vector(mod, CODE)
    out = Counter()
    for k in range(count):
        size, q = leb(body, q)
        end = q + size
        groups, q = leb(body, q)
        for _ in range(groups):          # local declarations
            _, q = leb(body, q)
            q = valtype(body, q)
        while q < end:
            op, q = instr(body, q)       # the opcode and every immediate
            out[op] += 1
        if q != end:
            raise ValueError(f"function body {k}: decoded past its end")
    return out

the producers answer imports and their count, imported functions and modules, exports and exported functions, memory pages, maximum and bytes, whether the memory is imported or exported, table size and maximum, the functions an element segment places, call and indirect-call counts, the start function and the data segment count, each also as a role field (dNN_analysis_<role>_<name>). a member that is not webassembly keeps the reading it had before. a module's function count is its own functions, then its own plus the imported ones as the second candidate.

a recorded stand-in. when a dynamic-trace tag is not in the known table and a sandboxed run of the build answers instead, the run now records the missing row, and the regression replay fails on it (the table check).

day 17 additions

the next keys of each record

every tag was asked two more fields on day 17 (decoder ring): the next two keys of its analysis record, alphabetical inside the configuration group. the producers below answer them.

the storage form of the configuration blob

the carrier search already knew which form found the blob (raw, base64, hex, utf-16, xor'd, compressed); it now keeps that stage with the hit, and the storage label is read from that stage and the member's section table at the blob's offset ([redacted]). a form no question has graded comes back low-confidence.

section = region_at(holder, carrier.offsets[0]).name
return label_for(carrier.stage, section)        # [redacted]

the organizer then rewrote the two storage questions: each now lists the allowed forms and a rule for each, in order (an encoded carrier, a section of the blob's own, a webassembly data segment, another native data section, else plain). the producer follows that order and reads the question when it carries the list: the one listed form that names the blob's place wins, and a rule that limits a form to writable data sends a blob in read-only data on to the next form. every region now carries its section's writable and executable flags (elf, pe and mach-o headers).

region = region_at(holder, at)                  # name, writable, executable
if not region.writable and writable_only(question_rules):
    return [next_form, ...]                     # [redacted]

c++ names on a pe build

typeinfo_symbol, typeinfo_name_symbol, vtable_symbol and mangled_<word>_symbol name mangled symbols (the class with a vtable first). a mingw pe lists a comdat member under more than one name in its coff symbol table; on a pe the [redacted] form ranks first.

a question that names the form decides the order: one that asks for the bare mangled name puts it first, one that names the coff section prefix puts the prefixed name first, and with neither the graded order stays.

runtime markers and a class's access word

has_go_runtime (go's build-info magic or runtime symbols), has_rust_mangled_symbol (a legacy or v0 rust symbol) and has_coroutine_runtime (kotlin's coroutine classes) answer true or false; access_flags is the jar main class's access word.

day 19 additions

day 19 placed the validated hcfg record inside a packed overlay (jack in the box). the carrier search now treats an appended overlay as another way a record can be stored.

finding the container boundary

the preferred overlay location comes from the executable structure. for an elf it is the end of the section-header table; for a pe it is the greatest raw end of any section. the parser checks for a four-character uppercase magic there before falling back to the last candidate whose declared packed length reaches the end of the file. the fallback supports zip-based members and other containers without teaching the overlay reader every file format.

the 44-byte header holds a version, two byte keys, a little-endian packed length and the sha256 of the unpacked bytes. a nonzero length must end at the member boundary, or at the structurally computed boundary plus a complete overlay.

proving the unpacking step

known magic values map to a short sequence of byte operations. each operation is a 256-byte translation table: xor, addition or subtraction by either key; left or right rotation; or complement. composing two operations means indexing the second table with every byte of the first:

def compose(first, then):
    return bytes(then[value] for value in first)

candidate = packed.translate(compose(xor_key2, ror3))

when the magic is new, a breadth-first search tries compositions up to three operations. every candidate is tested against the digest in the header. a match records the operation sequence for the rest of the run; no match raises an error naming the magic, keys and operations available so a question-specific sequence can be registered directly.

feeding the existing record parser

the unpacked bytes re-enter the ordinary carrier stages (raw, base64, hex, wide string and single-byte xor). the first validated hcfg hit retains both offsets: the outer overlay's file position and the record's position inside the unpacked payload. producers expose the overlay offset, total overlay size, magic and unpacked sha256, while configuration producers keep using the same crc-checked json record as earlier days.

day 20 additions

a header with scratch after the data

the RS20 overlay (resourceful) puts the resource at +48 and appends scratch bytes after it, with the scratch length as a u16le at +12 and the scratch's sha-256 at +16..+47. the layout model gained a scratch field: an overlay is accepted when its data ends the member or when data plus scratch does, and the scratch bytes travel with the overlay.

the question's words for the layout

the layout parser reads the question sentence: "resource length u32le at +8", "scratch length u16le at +12", "the resource starts at +48" and the "+16..+47 are the scratch sha-256" range, each word's gap to its offset bounded by a comma or a semicolon (the sha-256 clause used to swallow the next one). new producers read the proven overlay: the resource length and offset, the scratch length and its sha-256.

day 21 additions

role pairs across extensions

day 21's tag held an ocaml loader and the .cmxs plugin it loads (shared custody) and asked loader_machine, plugin_machine and the plugin's configuration under plugin_-prefixed names. the role rule (day 15: a word of the field names one member, which has a sibling differing by that word alone) compared whole file names, and -loader.elf and -plugin.cmxs differ in extension too, so no pair was found. siblings are now compared on stems:

stem = without(Path(member).stem, word)
siblings = [f for f in files if f != member
            and any(without(Path(f).stem, w) == stem for w in stem_words(f))]

an extension word still names no role, and a lone member is still never one.

day 26 additions

a jar's crash dump

day 26's tag was a groovy jar that stores its own crash dump as a json member beside the classes (crash course), and two new fields read it: the fault method, the first string of the dump's frames array, and whether every frame is a method of the main class. the producers for crash_* fields read the dump through one function: it takes the one member whose path looks like a crash or trace json, parses it and normalizes the keys to lowercase snake case. no such member, more than one, or a json that is not an object raise rather than pick.

CRASH_MEMBER = re.compile(
    r"(?:^|/)(?:crash|dump|trace|stack)[^/]*\.json$|(?:^|/)crash/[^/]+\.json$", re.I
)


def crash_trace(jar: bytes) -> dict[str, Any]:
    z = open_zip(jar)                         # bounded reads
    hits = [n for n in z.namelist() if CRASH_MEMBER.search(n)]
    if len(hits) != 1:
        raise ValueError(f"crash_*: {len(hits)} crash json members in the jar")
    found = json.loads(read_member(z, hits[0]))
    if not isinstance(found, dict):
        raise ValueError(f"crash_*: {hits[0]} is no json object")
    return {re.sub(r"[^a-z0-9]+", "_", str(k).lower()).strip("_"): v
            for k, v in found.items()}

the frames are listed innermost first, the order a java stack trace prints, so the producers are positions and a set comparison:

field names value
crash_fault_method, crash_faulting_method, crash_top_frame, crash_frame the first frame
crash_entry_method, crash_bottom_frame, crash_root_frame the last frame
crash_frame_count, crash_depth, crash_stack_depth and spellings the number of frames
crash_frames, crash_stack, crash_stack_frames the frames as compact json
crash_frames_match_methods and three spellings true when every frame is in the main class's method table
crash_kind, crash_exception, crash_type and spellings the dump's kind

the method-table comparison uses the class-file parser from day 10: it reads the manifest's Main-Class, parses that class and its package's classes, and the main class's method names are its table, including the constructor and whatever the compiler added (groovy adds its metaclass accessors). any other crash_<key> field reads that key of the dump by name, a list as compact json and a boolean as its lowercase word, and a key the dump lacks raises with the keys it has.

day 28 additions

questions read as recipes

day 28's cumulative bundle asked 32 field names our producers had never answered, most of them new questions about the tags of earlier days (family resemblance). the producers answer a field by its name, and a name like has_nim_runtime on an old tag matched none. the questions spelled their own rules, though, so the readers now parse a question into a recipe before they look at its name: a selector for which members to read and an operation for what to compute.

SELECT_ENDS = re.compile(
    r"\bUse the unique d(\d{2})-+ member whose basename ends "
    r"([\w.+-]+?)(?=[.;,]?(?:\s|$))"
)
CONTAINS = re.compile(
    r"\b(?P<truth>true|false) exactly when the final bytes"
    r"(?: interpreted as (?P<encodings>[\w-]+(?: or [\w-]+)*))?"
    r" contain(?: the case-sensitive(?: (?P<encoding>[\w-]+))? text)?"
    r" (?P<texts>.+?)"
    r"(?: or any nonempty ELF symbol name contains (?P<symbol>.+?))?"
    r"(?=\.(?:\s|$)|$)"
)

the selectors are every member of a tag in archive order, the unique member whose basename ends a given suffix, the primary managed assembly that a runtime configuration file sits beside, and a named pair whose decoded configurations must agree. the operations are a literal test over the member's bytes in one or two encodings (with an optional clause over symbol names, and a "false exactly when" negation), a count of symbol names matching the question's own regular expression, a sha-256 over a sorted or ordered list joined with the stated separator and trailing rule, a header field by its spec name rendered as the question says, an opcode count over a named section in one left-to-right scan, the packed overlay's one-byte keys, the container format and machine enums, the cli runtime, a jar main class's this_class, a configuration property, and counts, lists and tests over a member's named tables (sections, symbols, dynamic symbols, exports, imports, fields, methods, jar entries). a sentence that matches no shape has no recipe and goes to the name-based producers as before. the timed modes prefix each question with its delivery ("in delivery 3: ..."); that prefix is removed before parsing, so every set of a mastery shares one parse per question.

readers that dispatch on the member's format

the rules first written for the 32 fields called elf readers directly, and the timed sets shipped some older tags as pe builds under the same tag and the same question text. every recipe operation now dispatches on the member's own magic: symbol names from an elf symbol table or a pe's coff symbols, a section's bytes from either section table, header fields from the struct of the member's real format (an elf field asked of a pe reads the pe field of the same meaning).

def symbol_names(b: bytes) -> tuple[str, ...]:
    if b[:4] == b"\x7fELF":
        return elf_symbol_names(b)
    if file_format(b) == "pe":
        return coff_symbol_names(b)
    raise ValueError(f"symbol names: a {file_format(b)} member has none")

the family and its false friend

two members whose decoded configuration objects are equal are a family, and a member whose valid configuration differs is the false friend. the cluster is now a producer: the family's members, their count and the outlier, by name or by position, whichever form a question asks for; a configuration property of the family is read only when the pair agrees byte for byte, and a pair that differs is an error rather than a guess.

day 30 additions

a configuration record under a renamed magic

day 30's three neko .n members each ended in a configuration record (campaign trail): the hcfg layout (byte 4 a version, byte 5 an xor key, a little-endian 16-bit length, a crc32 over the decoded json) under the magic HGRF. both readers scanned for the literal HCFG, so eight fields raised no valid hcfg blob, though each question spelled the rule: the member (basename ends config.n, loader.n), the magic and a dotted json path. a record is now found under any printable four-byte magic: byte i+12 xor byte i+5 is the json's opening brace under its key, found in one pass over the member read as a big integer, and the crc decides (0.15s on a 64MB elf, no false hits).

def crib_starts(b: bytes) -> list[int]:
    n = int.from_bytes(b, "big")
    diff = (n ^ (n >> 56)).to_bytes(len(b), "big")
    out, j = [], diff.find(b"{", 12)
    while 0 <= j < len(b) - 1:
        i = j - 12
        if b[j + 1] ^ b[i + 5] in RECORD_SECOND:
            out.append(i)
        j = diff.find(b"{", j + 1)
    return out

the recipe reader takes the member, the magic and the path from the question's sentence, reads the last valid record under the named magic (else under any magic, and says so on stderr) and walks the path on the decoded dict. the loader's questions also ask to confirm that all three graphs match when this_role is omitted; that comparison is now made as stated, and a mismatch is an error naming the first differing key and the two members.

every carrier stage without a magic

the carrier search (raw, base64, hex, utf-16, single-byte xor, compressed streams, packed overlays, a jar's tail) anchored on the letters of the old magic, so a new magic inside any carrier read nothing. text carriers are now decoded and then scanned, an xored carrier is read by the same crib (the outer key cancels in the xor of two of its bytes), compressed and overlay data carry the any-magic scan inside, and the bytes past a jar's end of central directory are one more view. a matrix of magic, carrier and member kind reads every cell with no false read, at worst 0.20s more on the 64MB elf.

producers for techniques never asked by name

the day-30 prompt audit (replay and analysis) found earlier binary prompts whose technique had no producer by name and no recipe source word. each is now a producer that reads the member's real format and raises naming what it looked for, with recipe words so a spelled question reads the same lists (recipe.parse agrees with the previous tree on all 6393 binary questions):

objective-c   the gnustep runtime tables by section (classes, selectors,
              methods, constant strings), never by symbol: a stripped
              build keeps them
p/invoke      the .net ImplMap and ModuleRef tables, ahead of "import" so
              "P/Invoke import names" never reads the pe import directory
toolchain     compiler signatures in the bytes, most specific first; the
              elf .comment strings in file order
handshake     a symbol one member exports in .dynsym and another names as
              a whole string without defining it
call graph    an edge per direct, got and register call; callers_of_<name>,
              callees_of_<name>, edge counts, a joined sha-256 of callers
templates     every symbol through one c++filt run; an expansion's own
              qualified name names a template instance
bounds pair   day 25's guard, sentinel, faulting reader, index and trigger,
              read from the class files' constant pools, never executed

the call graph inside a timed set

the call graph of day 8's 67MB build took 7 to 14 seconds of a 15-second set. objdump now runs as 16 ranges cut at function starts, in parallel, and its output is read through a substring pre-filter; past a five-second budget the graph holds the ranges it read and every call-graph field goes out unsure. recipe.Digested keeps the graph and the disassembly by the member's sha256 across a mastery's sets: about 7.1 seconds down to 2.3, and a later set 0.02 to 0.04.

changelog

{generated by a clanker}