cross-lane field resolution: answering a field with another problem's parser
our solvers are organised by challenge category: a metadata lane answers file properties, a binary lane answers analysis fields over executables, an apk lane answers app fields, a script lane answers specimen fields. each lane has its own vocabulary of field names and its own readers. on day 12 a metadata question asked for a .net module version id. the reader that parses .net metadata tables belonged to the binary lane, and the metadata lane had no way to ask it. this layer lets any lane answer a field it cannot name with another lane's reader of exactly that name.
challenge posts that use it (from day 15 on, a metadata time trial or mastery set asks every earlier day's property again, so a field keeps reaching this layer after its own day):
dotnet-mvid-guid, read by the dotnet lane: metadata managed expectations (day 12, all three modes); asked again in the metadata time trial and mastery of a word to the wise (day 15), return to sender (day 16), hear me out (day 17), frame of reference (day 18), little endian on the prairie (day 19), universal appeal (day 20), package deal (day 21), main character energy (day 22), page me maybe (day 23), event horizon (day 24), hive mind (day 25), before you go (day 26), packet loss prevention (day 27), where it all began (day 28), trust issues (day 29).apk-package-name, read by the mobile lane: metadata package deal (day 21, all three modes); asked again in the metadata time trial and mastery of main character energy (day 22), page me maybe (day 23), event horizon (day 24), hive mind (day 25), before you go (day 26), packet loss prevention (day 27), where it all began (day 28), trust issues (day 29).jar-main-class, read by the binary lane: metadata main character energy (day 22, all three modes); asked again in the metadata time trial and mastery of page me maybe (day 23), event horizon (day 24), hive mind (day 25), before you go (day 26), packet loss prevention (day 27), where it all began (day 28), trust issues (day 29).
input model
a field name, the bytes of the member the question names, the member's file name, and the lane asking (whose own readers already failed and are skipped). field names arrive in each lane's spelling, dotnet-mvid-guid or module_mvid, so a name is first reduced to lowercase words split on -, _, . and spaces.
a leading word may be a family (pe, dll, dotnet, elf, so, macho, jar, apk, android, dex): it names the format the field is about. it is dropped before the name is matched against readers that already know their format, and it selects a member when the bytes are a container.
parsing or analysis algorithm
every reader is asked in a fixed order, each by its own exact vocabulary:
metadata the forensics property table, struct fields and version-resource
keys by name; never its header-field or exiftool guesses
binary the field-producer registry over executables (elf, pe, mach-o, jar)
dotnet the named metadata columns of a managed pe
apk the app fields of the manifest and network security config
def resolve(field: str, data: bytes, name: str, skip: str) -> Answer:
ws = words(field)
hits = _hits(ws, data, name, skip)
if not hits:
family, _ = _family(ws)
inner = _members(data, family)
if len(inner) == 1:
where, body = inner[0]
hits = _hits(ws, body, where, skip)
if len({repr(h.value) for h in hits}) > 1:
raise KeyError(f"crosslane: lanes disagree on {field!r}: ...")
if not hits:
raise KeyError(f"crosslane: no lane reads {field!r} by name; closest ...")
return hits[0].value
a reader that does not read the bytes (a pe reader handed a zip, an apk reader handed an elf) raises its own parse error, which counts as no answer from that reader. when nothing reads the bytes themselves and the field names a family, the one member of a zip or apk in that format is tried instead (an apk's classes.dex, a library's .so).
the dotnet reader exposes a managed pe's identity by ecma-335 names: every column of the one-row Module and Assembly tables (module-mvid, module-name, assembly-name, assembly-culture, ...), assembly-version (major.minor.build.revision), the assembly's string attributes (target framework, configuration, file version, informational version, company, product, title, ...), its references, and a column's bare name where only one table has it (mvid). a guid column may carry its kind as a trailing word (dotnet-mvid-guid); no other trailing word is accepted.
validation and ambiguity
- exact names only. a field that merely shares words with a reader is not answered: the miss names the closest readers instead, ranked by shared words, so a person or a solver change can map it in one line.
- an empty value is a miss, never an empty answer: an assembly without a culture does not answer
assembly-culturewith nothing. - two readers that answer the same name differently raise, naming both, so the question decides which one was meant.
- in the metadata lane the layer runs after the lane's own exact routes and before its guesses (raw header fields and exiftool tags matched by words): an exact reader of another lane outranks a guess of this one.
integration
each lane calls it where it would otherwise fail: the metadata property resolver before its guesses, the binary lane's generic producers for a name no producer has, the apk lane for an app field its manifest tables do not name, and the script lane for a field the specimen's descriptor does not answer. the printed trace names the reader that answered:
crosslane: 'dotnet-mvid-guid' read by the dotnet lane (dotnet module-mvid, managed-001.dll)
limitations
- a field the bytes answer but no reader names still needs a new reader; this layer only reuses what exists.
- names are matched after word splitting and one family word; synonyms (two names for one value) are not inferred.
- the member descent is one level deep and needs exactly one member of the family's format.
day 15 additions
exact names leave one gap open: a field whose value some reader already outputs under other words. day 14 asked for an lnk target path the lnk record held as target; day 15 asked for an ooxml document's external relationship targets, which the office reader held as external. both times the value was on hand and nothing connected the two names.
the reader inventory closes it. every reader that applies runs over every file of the set (the record readers, the office document reader, the header structs, elf, pe, apk and .net), and each output field becomes one reading: (file, reader, key, value). a reading answers a field when every word of its key is among the field's words. more key words rank first, then the file whose name shares the field's words (a number in the field picks the file whose name carries it):
def ranked(question, files, field):
fw = field_words(question, field)
scored = []
for r in readings(files):
kw = words(r.key)
if not kw or not kw <= fw:
continue
values = forms(question, r.value)
if values:
scored.append(((-len(kw), -member_score(r, fw)), r, values))
scored.sort(key=by_score)
return [(r, values) for _, r, values in scored]
forms writes a reading the way the question's answer kind allows: an integer field takes only digits, an enum only a listed value, and a list only a json field, as compact arrays in more than one order. at most eight values are tried per field. once one is confirmed, the inventory keeps which reader, key and written form produced it, and answers the same field on other files by reading that key again; a file without it is a miss, never a guess.
a field nothing answers fails with something to work from: the readings that share any word with it, most shared words first, each as the one call that reads it, with its value:
inventory.read(member(files, "<file>"), "office", "external") = '[...]' (shares ['external'])
day 21 additions
the package name across categories
the metadata category's apk-package-name (day 21, package deal) is the first apk field a non-mobile question asked. the resolver's android reader looks a field's words up in the android toolkit's named values, which now carry every manifest and signer scalar; the same values feed the reading inventory, so a field matched by its words, or learned from a value the grader confirmed, finds the manifest attribute without a reader written for it.
day 25 additions
readings one word over
the reader inventory (day 15) kept a reading only when every word of its key was among the field's words. day 25's registry-hive-root-name was the hive record's root_key_name (hive mind): the key has key and the field does not, so the inventory answered nothing, and its miss report listed exactly that reading first, with the right value.
after the exact readings, the inventory now keeps the readings whose key shares at least two words with the field and carries at most one word the field lacks, qualifier words not counted. an exact reading still outranks every near one; the near readings follow, most shared words first, then fewest extra words, then the file whose name the field's words name:
exact, near = [], []
for r in readings(files):
kw = words(r.key)
if not kw:
continue
over = kw - fw - QUALIFIERS
if not kw <= fw and (len(kw & fw) < 2 or len(over) > 1):
continue
values = forms(question, r.value)
if not values:
continue
if kw <= fw:
exact.append(((-len(kw), -member_score(r, fw)), r, values))
else:
near.append(((-len(kw & fw), len(over), -member_score(r, fw)), r, values))
exact.sort(key=by_score)
near.sort(key=by_score)
return [(r, values) for _, r, values in [*exact, *near]]
a key two words over, or one that shares a single word with the field, stays out. the resolver above still matches exact names only; the near tier belongs to the inventory, and the metadata record guesser has the same rule for record fields (archive and metadata readers).
day 30 additions
a miss report that always ends in a draft
a field nothing answers at release is handed over with the inventory's miss report (day 15): what was tried, the closest readings as the one call that reads each, the path of the bytes the field means, the set's files and the answer syntax. a reading that shared two words with the field also gave the report a draft of the field's rule. day 30's two misses had no such reading, so both reports ended with nothing to start from, and neither carried the question, though the binary question spelled out the whole algorithm: the record's magic, its layout, the member and a dotted json path (binary analysis). the person fixing it read bytes for about 43 seconds before finding the question.
the report now opens with the field's question text (up to 600 characters), carries more of the answer syntax, and always ends in a draft: the closest reading's call, else a skeleton whose member line names the file the question names (basename ends config.n, loader.n's, for cluster-001.zip), kept only when that line picks exactly one file of the set. the member line passes the name's words, since the member lookup matches whole words: a draft that named the whole file name matched no file.
other lanes' readers as relation keys
the metadata lane's relation family (archive and metadata readers) keys a member by what it holds, and two of those keys are another lane's reading: an apk by the android reader's package and signer digest, a managed pe by the .net reader's assembly identity (name, version, culture, public key token). the signer read runs openssl, and the android reader keeps its values per set for every caller; a caller with a time cap (the relation read's tool budget) now gets a view with its own cap over those shared values, and a signer read cut short by its cap is not kept, so the next field reads it again at its own cap instead of finding an empty signer.
changelog
- day 12: the resolver over the metadata, binary, dotnet and apk readers; named .net metadata columns, assembly attributes and references; hooks in the metadata, binary, apk and script lanes.
- day 15: the reader inventory: every reader over every file as readings, matched by key words within the field's words, written forms per answer kind, a confirmed reading reused on other files; a miss lists the closest readings as calls.
- day 21: the android reader's named values carry every manifest and signer scalar.
- day 25: readings one word over the field (two shared words or more, one extra at most) ranked after the exact readings.
- day 30: the miss report opens with the question and always ends in a draft whose member line names the file the question names; the apk signer and the .net assembly identity as relation keys, the signer read capped per caller.
{generated by a clanker}