cross-lane field resolution: answering a field with another problem's parser

our solvers are organised by challenge category: a metadata lane answers file properties, a binary lane answers analysis fields over executables, an apk lane answers app fields, a script lane answers specimen fields. each lane has its own vocabulary of field names and its own readers. on day 12 a metadata question asked for a .net module version id. the reader that parses .net metadata tables belonged to the binary lane, and the metadata lane had no way to ask it. this layer lets any lane answer a field it cannot name with another lane's reader of exactly that name.

challenge posts that use it (from day 15 on, a metadata time trial or mastery set asks every earlier day's property again, so a field keeps reaching this layer after its own day):

input model

a field name, the bytes of the member the question names, the member's file name, and the lane asking (whose own readers already failed and are skipped). field names arrive in each lane's spelling, dotnet-mvid-guid or module_mvid, so a name is first reduced to lowercase words split on -, _, . and spaces.

a leading word may be a family (pe, dll, dotnet, elf, so, macho, jar, apk, android, dex): it names the format the field is about. it is dropped before the name is matched against readers that already know their format, and it selects a member when the bytes are a container.

parsing or analysis algorithm

every reader is asked in a fixed order, each by its own exact vocabulary:

metadata   the forensics property table, struct fields and version-resource
           keys by name; never its header-field or exiftool guesses
binary     the field-producer registry over executables (elf, pe, mach-o, jar)
dotnet     the named metadata columns of a managed pe
apk        the app fields of the manifest and network security config
def resolve(field: str, data: bytes, name: str, skip: str) -> Answer:
    ws = words(field)
    hits = _hits(ws, data, name, skip)
    if not hits:
        family, _ = _family(ws)
        inner = _members(data, family)
        if len(inner) == 1:
            where, body = inner[0]
            hits = _hits(ws, body, where, skip)
    if len({repr(h.value) for h in hits}) > 1:
        raise KeyError(f"crosslane: lanes disagree on {field!r}: ...")
    if not hits:
        raise KeyError(f"crosslane: no lane reads {field!r} by name; closest ...")
    return hits[0].value

a reader that does not read the bytes (a pe reader handed a zip, an apk reader handed an elf) raises its own parse error, which counts as no answer from that reader. when nothing reads the bytes themselves and the field names a family, the one member of a zip or apk in that format is tried instead (an apk's classes.dex, a library's .so).

the dotnet reader exposes a managed pe's identity by ecma-335 names: every column of the one-row Module and Assembly tables (module-mvid, module-name, assembly-name, assembly-culture, ...), assembly-version (major.minor.build.revision), the assembly's string attributes (target framework, configuration, file version, informational version, company, product, title, ...), its references, and a column's bare name where only one table has it (mvid). a guid column may carry its kind as a trailing word (dotnet-mvid-guid); no other trailing word is accepted.

validation and ambiguity

integration

each lane calls it where it would otherwise fail: the metadata property resolver before its guesses, the binary lane's generic producers for a name no producer has, the apk lane for an app field its manifest tables do not name, and the script lane for a field the specimen's descriptor does not answer. the printed trace names the reader that answered:

crosslane: 'dotnet-mvid-guid' read by the dotnet lane (dotnet module-mvid, managed-001.dll)

limitations

day 15 additions

exact names leave one gap open: a field whose value some reader already outputs under other words. day 14 asked for an lnk target path the lnk record held as target; day 15 asked for an ooxml document's external relationship targets, which the office reader held as external. both times the value was on hand and nothing connected the two names.

the reader inventory closes it. every reader that applies runs over every file of the set (the record readers, the office document reader, the header structs, elf, pe, apk and .net), and each output field becomes one reading: (file, reader, key, value). a reading answers a field when every word of its key is among the field's words. more key words rank first, then the file whose name shares the field's words (a number in the field picks the file whose name carries it):

def ranked(question, files, field):
    fw = field_words(question, field)
    scored = []
    for r in readings(files):
        kw = words(r.key)
        if not kw or not kw <= fw:
            continue
        values = forms(question, r.value)
        if values:
            scored.append(((-len(kw), -member_score(r, fw)), r, values))
    scored.sort(key=by_score)
    return [(r, values) for _, r, values in scored]

forms writes a reading the way the question's answer kind allows: an integer field takes only digits, an enum only a listed value, and a list only a json field, as compact arrays in more than one order. at most eight values are tried per field. once one is confirmed, the inventory keeps which reader, key and written form produced it, and answers the same field on other files by reading that key again; a file without it is a miss, never a guess.

a field nothing answers fails with something to work from: the readings that share any word with it, most shared words first, each as the one call that reads it, with its value:

inventory.read(member(files, "<file>"), "office", "external") = '[...]' (shares ['external'])

day 21 additions

the package name across categories

the metadata category's apk-package-name (day 21, package deal) is the first apk field a non-mobile question asked. the resolver's android reader looks a field's words up in the android toolkit's named values, which now carry every manifest and signer scalar; the same values feed the reading inventory, so a field matched by its words, or learned from a value the grader confirmed, finds the manifest attribute without a reader written for it.

day 25 additions

readings one word over

the reader inventory (day 15) kept a reading only when every word of its key was among the field's words. day 25's registry-hive-root-name was the hive record's root_key_name (hive mind): the key has key and the field does not, so the inventory answered nothing, and its miss report listed exactly that reading first, with the right value.

after the exact readings, the inventory now keeps the readings whose key shares at least two words with the field and carries at most one word the field lacks, qualifier words not counted. an exact reading still outranks every near one; the near readings follow, most shared words first, then fewest extra words, then the file whose name the field's words name:

exact, near = [], []
for r in readings(files):
    kw = words(r.key)
    if not kw:
        continue
    over = kw - fw - QUALIFIERS
    if not kw <= fw and (len(kw & fw) < 2 or len(over) > 1):
        continue
    values = forms(question, r.value)
    if not values:
        continue
    if kw <= fw:
        exact.append(((-len(kw), -member_score(r, fw)), r, values))
    else:
        near.append(((-len(kw & fw), len(over), -member_score(r, fw)), r, values))
exact.sort(key=by_score)
near.sort(key=by_score)
return [(r, values) for _, r, values in [*exact, *near]]

a key two words over, or one that shares a single word with the field, stays out. the resolver above still matches exact names only; the near tier belongs to the inventory, and the metadata record guesser has the same rule for record fields (archive and metadata readers).

day 30 additions

a miss report that always ends in a draft

a field nothing answers at release is handed over with the inventory's miss report (day 15): what was tried, the closest readings as the one call that reads each, the path of the bytes the field means, the set's files and the answer syntax. a reading that shared two words with the field also gave the report a draft of the field's rule. day 30's two misses had no such reading, so both reports ended with nothing to start from, and neither carried the question, though the binary question spelled out the whole algorithm: the record's magic, its layout, the member and a dotted json path (binary analysis). the person fixing it read bytes for about 43 seconds before finding the question.

the report now opens with the field's question text (up to 600 characters), carries more of the answer syntax, and always ends in a draft: the closest reading's call, else a skeleton whose member line names the file the question names (basename ends config.n, loader.n's, for cluster-001.zip), kept only when that line picks exactly one file of the set. the member line passes the name's words, since the member lookup matches whole words: a draft that named the whole file name matched no file.

other lanes' readers as relation keys

the metadata lane's relation family (archive and metadata readers) keys a member by what it holds, and two of those keys are another lane's reading: an apk by the android reader's package and signer digest, a managed pe by the .net reader's assembly identity (name, version, culture, public key token). the signer read runs openssl, and the android reader keeps its values per set for every caller; a caller with a time cap (the relation read's tool budget) now gets a view with its own cap over those shared values, and a signer read cut short by its cap is not kept, so the next field reads it again at its own cap instead of finding an empty signer.

changelog

{generated by a clanker}