tooling: archive and metadata analysis
metadata challenges look simple until archives nest, filenames rotate, or an answer id names a logical artifact rather than an exact member. our toolkit combines lossless extraction, evidence-based member resolution, and a registry of byte properties.
extraction without losing identity
raw downloads are always retained. zip-family downloads are also expanded, and nested archives are placed beneath a path derived from the parent name. member iteration preserves archive order because some questions ask for the first or nth member. containers and extracted members remain distinguishable.
encrypted winzip-aes archives are delegated to 7-zip; ordinary zips are read in memory. the common malware-analysis password infected is supplied explicitly when required.
resolving the member named by a field
fields commonly have the form <member-stem>-<property>. the resolver uses several independent clues, in order:
- an exact artifact basename from the question, if that member exists.
- an exact basename stem parsed from the answer field.
- a whole-token numeric identifier from the stem.
- the sole non-container member, if only one exists.
this is intentionally stricter than substring matching. 001 must not match 1001, and a field containing text must not be split at the wrong -text-.
property registry
the parser matches the longest known suffix and dispatches it to a function of the member bytes and basename:
PROPERTIES = {
"filename": lambda data, name: name,
"size-bytes": lambda data, name: str(len(data)),
"md5": lambda data, name: hashlib.md5(data).hexdigest(),
"sha1": lambda data, name: hashlib.sha1(data).hexdigest(),
"sha256": lambda data, name: hashlib.sha256(data).hexdigest(),
"text-utf8-bom-present": utf8_bom_present,
}
magic-prefix-hex-N is parameterized: it returns data[:N].hex(). boolean properties return lowercase true or false because exact answer vocabulary is part of the format.
text properties
byte-order marks are checked before decoding, with utf-32 markers tested before utf-16 because the little-endian forms share a prefix. the toolkit can also describe likely encoding, utf-8 validity, newline style, line count, character count, trailing newline, and nul presence.
these values come from raw bytes. decoding and re-encoding first could consume a bom, normalize newlines, or otherwise destroy the property being measured.
file signatures
a separate magic-byte detector recognizes common formats including elf, pe, zip, png, pdf, jpeg, gif, gzip, 7-zip, rar, dex, and mach-o. this deliberately ignores extensions. an extension is useful context, but the delivered bytes are the evidence.
day 3 additions
prepared format readers
on day 3 we also wrote readers for formats past years used, so a new artifact family would start from parsed structure rather than raw bytes: packet captures (summary, field extraction, usb keystrokes, icmp data, http export), windows event logs (powershell script blocks, process creation, services, logons), registry hives (walk, search, persistence keys, userassist), office documents (macros, auto-exec, external targets, broken-zip repair), email, pdf (risky keys, inflated streams, javascript, uris), media (exif, qr, trailing data, png chunks, dtmf tones) and windows artifacts (lnk, compiled python, compressed streams, memory images). a magic-byte sniffer routes files to them. none has been needed by a graded field yet.
day 4 additions
jpeg frame dimensions
the property jpeg-sof-dimensions-pixels walks the jpeg marker stream: skip fill bytes, skip standalone markers (including restart markers), bounds-check every segment length, and stop at the first start-of-frame marker (0xc0-0xcf except 0xc4, 0xc8, 0xcc). the frame stores height and width as big-endian 16-bit values, serialized in the grader's form ([redacted]). like every other property it is one registry entry, so stems and mastery sets need no new code.
image dimensions in any common format
after the jpeg question we generalized the property. image_size reads the coded header of each format directly: png's IHDR (big-endian width and height at bytes 16-23), gif's logical screen (little-endian at 6-9), bmp's info header (little-endian at 18-25, where a negative height means top-down rows), webp's VP8X/VP8 /VP8L chunks (24-bit or 14-bit fields, some stored minus one), and jpeg's start of frame as above.
a field whose property is not registered yet but ends in dimensions-pixels, width or height resolves through a small fallback table. stems so far end in a number token (d04-jpeg-001), so the field is split after the last number and the remainder must match a fallback pattern exactly; without a number, the shortest matching tail is used. an unknown property that matches nothing still fails loudly.
day 5 additions
png dimensions come from the IHDR chunk at a fixed offset, through the same header reader as day 4, and the toolkit walks the full chunk list:
while i + 12 <= len(data):
size, kind = struct.unpack_from(">I4s", data, i)
body = data[i + 8 : i + 8 + size]
(crc,) = struct.unpack_from(">I", data, i + 8 + size)
chunks.append(PngChunk(kind.decode("latin-1"), body,
crc == zlib.crc32(kind + body) & 0xFFFFFFFF))
i += 12 + size
if kind == b"IEND":
break
trailing = data[i:] # anything after IEND
from that walk it reports chunk order and types, chunk types outside the png specification, failed crcs, the number of bytes after IEND, text chunks, and a geometry check. for a non-interlaced image the inflated IDAT stream must be height * (1 + ceil(width * channels * bit_depth / 8)) bytes, so a header that lies about the size disagrees with its own pixel data. the dimension reader still accepts any image format for a dimension field, so a mislabeled member still answers.
day 6 additions
a pdf-header-version property reads the %PDF- line in the form the grader expects ([redacted]), searched in the first kilobyte to tolerate leading junk, and none when absent. readers for catalog and effective version, eof markers, object and page counts, encryption and info-dictionary keys sit alongside it.
two general changes:
- splitting an unknown field. field ids are
<stem>-<property>, and the boundary used to come from the longest known property name, so an unknown name crashed before the member was even resolved. stems in this lane end in a number token, so an unrecognized tail now splits after the stem's last number. - a last resort for unknown properties. the property lookup runs exiftool once per member (cached, five-second timeout, failures reported by name) and picks the tag whose camel-case words overlap the property's words: it must contain the property's last word, or share at least two; a tie is never guessed. exiftool's
-args -ntext keeps values verbatim where its json output coerces some to numbers. replayed against the metadata fixtures of days 1-6 it never produced a wrong value.
zip-based files are identified by their marker members rather than reported as plain zips: an android manifest (apk), a root .nuspec (nuget, checked before office because both carry [Content_Types].xml), an extension.vsixmanifest, a mimetype entry naming epub or opendocument, [Content_Types].xml (office), a .dist-info/WHEEL (python wheel), a jar manifest or class files, and a manifest.json with manifest_version (browser extension). marker files are read only when their declared size is small, so a hostile archive cannot force a large decompression just to be identified. anything else stays a zip.
day 7 additions
raw header fields, matched by name
days 4-7 each graded one structural header field of a new format (jpeg start-of-frame dimensions, png IHDR dimensions, the pdf header version, a zip's end-of-central-directory entry count), and each needed a reader written under the clock. exiftool names many fields, but rarely the raw ones.
a header registry now parses the raw headers of gzip, tar, bmp, gif, png, riff (wav and webp), sqlite, id3, zip and pdf into fields named <struct>.<field>, the way the questions name them (zip.eocd_total_entries, pdf.header_version):
def gzip_fields(data):
method, flags, mtime, xfl, os_ = struct.unpack_from("<BBIBB", data, 2)
return {"header.method": str(method), "header.mtime": str(mtime),
"header.os": str(os_), ...}
an unknown property is matched to a parsed field before exiftool is asked: the candidate must share the property's last word (version, count, mtime) and the most words overall; a tie is never guessed. a few synonyms bridge question wording (timestamp -> mtime, bpp -> bits_per_pixel). with the hand-written readers for days 5, 6 and 7 removed, their fixtures still solve from these fields.
zip census
zip-eocd-* exposes every end-of-central-directory field; the central directory record count, local header count, file and directory entries, empty and non-empty files, and a claimed-versus-held mismatch flag are readers too. the central directory is walked from the eocd's own offset, so a lying entry count does not shape the walk.
nested archives as members
a field can name a zip inside the delivered bundle. member resolution matches the question's artifact text against every delivered name, archives included, instead of only leaf files.
day 8 additions
central directory names
zip-central-directory-member-names is the list of file names in central directory order, each with its directory prefix as stored, serialized as the question asks ([redacted]). it uses the same walk as the census (from the eocd's offset, stopping at the first record without PK\x01\x02). a twin property reads the names from the local file headers in file order, and the walk also gives total compressed and uncompressed bytes over every record (exiftool reports only the first member's size).
a guess must cover every word
the header-field and exiftool fallbacks used to accept the candidate that shared the property's last word and the most words overall. that answered png-file-size-bytes from a png header field with 0, and a zip total from the first member alone. a candidate must now cover every word of the property, through the field name or a synonym (timestamp for mtime), with filler words such as "value", "total" or "number" ignored; otherwise the solver raises an error that names the property. properties a family prefix makes predictable are explicit instead: <kind>-sha256, <kind>-md5, <kind>-file-size-bytes and <kind>-filename for any member, the pdf startxref offset, and png-<type>-chunk-count per chunk type.
bounded decompression
every reader that inflates data (archives, decoders, apk, office, jar, pdf, png IDAT, media and windows artifacts) goes through one bounded layer. an archive tree has a 512 mb checkpoint and a single stream a 64 mb one; past a checkpoint, expansion continues only while the output-to-input ratio stays at or under 100:1, up to a 4 gb ceiling, and nesting stops at depth four. real large data opens in the same pass without re-expanding what was already read, and a bomb ends as an error instead of exhausting memory. 7z runs under a file-size limit for the same reason.
day 9 additions
header fields by structured name
a property of the form <family>-<struct>-<field>-<width>-<format>, such as pe-coff-machine-u16-hex, elf-header-e-machine-u16-hex or zip-central-directory-crc32-u32-hex, needs no reader of its own. it resolves against tables of 23 header structs. each table lists field offset size rows and comes with a function that finds the struct's base offset and byte order in the bytes:
- pe: dos header, nt signature, coff file header and optional header. the optional header's magic picks the pe32 (
0x10b) or pe32+ (0x20b) layout, soimage_baseis four bytes in one and eight in the other. the nt, coff and optional headers are located throughe_lfanewat0x3c, after checkingMZandPE\0\0. - elf:
e_identand the file header, 32- or 64-bit byEI_CLASS, little- or big-endian byEI_DATA. - mach-o: the thin header (its four magics give byte order and word size) and the fat header. a fat magic shares
CAFEBABEwith java class files, so a value of 45 or more wherenfat_archsits is a class-file version and is refused. - gzip header and trailer, png
IHDR, jpeg start-of-frame and jfifAPP0, zip end-of-central-directory, central and local records, the pdf header version as two numbers, bmp file and dib headers (including the 12-byte os/2 core header with 16-bit sizes), the gif logical screen, and wavRIFF,fmtanddata.
the name is parsed before any byte is read. the tail gives signedness, width, an optional byte order and the output form:
STRUCT_TAIL = re.compile(r"-([ui])(8|16|32|64)(le|be)?-(hex|dec|int)$")
the tokens before it split into family, struct and field with separators ignored, so mach-o and macho, or e-machine and e_machine, name the same thing. each field is listed under its specification name and its plain words (time_date_stamp timestamp), and a field is never matched on part of its name. the value is then read from every record of the struct:
for rec in s.records(data):
offset, size = rec.layout[field]
if size != ref.size:
raise KeyError(f"{prop!r}: {s.family} {s.label}.{field} is {8 * size}-bit")
if ref.order not in (None, rec.order):
raise KeyError(f"{prop!r}: {s.family} {s.label} is {rec.order}-endian")
raw = rec.data[rec.base + offset : rec.base + offset + size]
if len(raw) != size:
raise ValueError(f"{s.family} {s.label}.{field} runs past the member end")
values.add(int.from_bytes(raw, rec.order))
hex is serialized in the grader's form ([redacted]); dec and int are decimal, signed for an i width.
loud failures
a name that does not fit the bytes is an error that says what the tables have, never a guess:
- an unknown family, struct or field; the error lists the structs the family has, or the fields the struct has.
- a width that is not the field's own (
pe-coff-machine-u32-hex), or a field the image's layout lacks (base_of_dataasked of a pe32+ image). - an explicit byte order the file contradicts (
elf-header-machine-u16be-hexon a little-endian elf). - bytes of another format: a gzip renamed to
.exehas noMZ, and a thin mach-o has no fat header. - records that disagree: zip local headers carrying different values for the asked field.
a spelling the tables lack is one line in a solver, forensics.struct_alias("<name>", "<known name>"), which answers the new name from the known field. field ids are split by taking the shortest tail of at least five tokens that names a known struct field, so the member stem no longer needs a number token: drop-mach-o-header-cputype-u32-hex splits after drop.
day 10 additions
shorter structured names
the day-9 resolver required <family>-<struct>-<field>-<width>-<format>. day 10 asked for pe-address-of-entry-point-rva-hex: no struct, no width, and a word (rva) saying what kind of value the field holds. the resolver now reads that form too:
- the struct may be left out when exactly one field of the family has the field's words; a name that two structs share raises.
- a last kind word is removed only when the field it leaves is of that kind:
rvaforaddress_of_entry_point,base_of_code,base_of_data;vaforimage_baseand the elfe_entry.pe-image-base-rva-hexraises. - a name with no width reads the field at its own width.
a name that matches nothing lists the closest fields the tables have, so a near miss names its neighbour in the error instead of only failing.
the pe version resource
hlib.versioninfo walks the resource directory to RT_VERSION and reads VS_FIXEDFILEINFO, every StringFileInfo table and the VarFileInfo translations, in pe32 and pe32+ of any machine. a string property is named by its key's words (pe-original-filename, pe-version-info-company-name), optionally with the scope (string-file-info, fixed-file-info) and the table (040904b0); numbers are struct names like any other (fixed_file_info, translation). where a property exists in both blocks, such as the file version, the string and the fixed block's dotted version must agree, or the read raises: a version resource that contradicts itself is the kind of lie a question may ask about, and the question's scope has to pick the source.
executable labels and zip provenance
for prompts about decoys and impossible origins, two families of properties are prepared and answer only when a question names them:
- executable labels: architecture, format, bits, target os, platform, the true format sniffed from the bytes, whether it can execute, and whether the extension matches the bytes. they are returned as ranked candidates, and a machine, os or extension the tables do not know raises. generic data extensions (
.bin,.dat,.raw, ...) claim no format, so they match only bytes that no reader names.cafe babeis split as libmagic splits it: a small architecture count is a universal mach-o, a large one a java class. - zip provenance: the creating host os and version, the version needed, and per-entry checks (path traversal, absolute paths, backslash separators, drive letters, encryption, data descriptors, unix modes from a non-unix host, dos timestamps that disagree with the utc extra fields), each as a flag, a name list and a count, plus duplicate names, case collisions and the archive comment.
day 11 additions
the rest of the pe
days 9 to 11 each asked one pe property (the coff machine, the entry point, a version string). the tables a pe has beyond those are now named properties too, answered only when a question asks for them:
- sections: names in table order (long
/Nnames resolved through the coff string table), the count, and the section that holds the entry point. - imports and exports: dll names as stored and lowercased, dll, function and total counts, the import hash, and export names.
- the subsystem, the link timestamp as utc and as an epoch, the pdb path from the debug directory, whether the file is a dll, a .net assembly, signed, or carries a rich header or a version resource.
- the checksum as stored and as computed, and whether they agree; the overlay's size and offset, and whether there is one.
lists are compact json in table order; a label no question has graded goes out as ranked candidates.
comparing a pe's names
pe-<a>-matches-<b> (also mismatch, mismatches, differs-from) compares two names a pe carries: the member's own file name, or a version string such as OriginalFilename or InternalName. the comparison ignores case, as windows file names do, and an extension-less side is compared by stem. resolving such a name exposed a bug: the generic property suffix (-name, -sha256, ...) had answered pe-internal-name-matches-original-filename with the member's name. a generic suffix now applies only directly after a family word.
archives
winzip-aes evidence zips are read in memory, through the same bounded member reader as plain zips, instead of being extracted to a temporary directory by an external tool; the tool remains the fallback for zipcrypto, unusual names and a bad password or mac. a member's zip index is now parsed once and shared by every reader that asks for it (format detection, jar checks, member search), where each used to parse it again.
day 12 additions
.net identity by name
a managed pe's identity is in its metadata tables, not its headers. the property resolver now hands any name it cannot answer exactly to cross-lane field resolution before it guesses, and the .net reader there exposes the Module and Assembly rows by column name, the assembly version, the assembly's string attributes (target framework, file and informational versions, configuration, company, product, title) and its references. dotnet-mvid-guid resolves to the module's version id from the #GUID heap; managed expectations walks through the tables.
day 13 additions
windows installer databases
an .msi is a relational database inside an ole2 compound file: one stream per table, and two streams, _StringPool and _StringData, that hold every string the tables use. the stream names are packed into private-use code points (two characters per code point from 0x3800, behind a 0x4840 marker), so the reader encodes the name it wants before it opens a stream:
def stream_name(name: str) -> str:
out = "\u4840"
i = 0
while i < len(name):
c1 = ALPHABET.index(name[i])
if i + 1 < len(name):
out += chr(0x3800 + c1 + (ALPHABET.index(name[i + 1]) << 6))
i += 2
else:
out += chr(0x4800 + c1)
i += 1
return out
the string pool's header carries the codepage and whether table cells are two or three bytes wide; the Property table is two string columns stored one after the other. any msi-<property words> field reads the row whose name has those words (ProductCode, UpgradeCode, ProductVersion), with a shape word such as guid dropped first, and falls back to the summary information stream, where the package code is the revision number. product placement walks through one installer.
fields across members
some questions name several files and ask for a fact about the set: "compute the full-file sha-256 digest of each named file, then submit a compact json array of their filenames in ascending digest order. files: [...]". the question carries everything: the file list, the per-file property and the ordering. the reader takes the property from the field name and the question's words (a digest, the size, or any property the lane reads), computes it per named member, and returns the ordered list or the one file at either end:
names = [str(n) for n in json.loads(listed.group(1))]
prop = _prop(field, text)
values: dict[str, Any] = {}
for name in names:
stem = PurePosixPath(name).stem
_, body = archive.resolve_member(files, stem, {"artifact": f"x -> {name}"})
values[name] = _value(prop, name, body)
def key(name: str) -> tuple[Any, str]:
return values[name], name
ordered = sorted(names, key=key, reverse=descending)
if verb in ("order", "sorted", "ranking", "rank") or "array" in low:
return json.dumps(ordered, separators=(",", ":"))
return ordered[0]
day 14 additions
one-record readers, mapped by the property's words
day 14 asked a property of a windows shortcut. the toolkit already parsed shortcuts; nothing mapped lnk-target-path-windows to the parsed target. now every family with a record reader answers any property by its words: the reader returns one flat record ({field: value}), both the property and the field names are spelled one way (modification is write, id is guid, network is net), qualifiers that only say how a value is written (path, windows, hex, utc) are set aside, and the field whose words the property covers most wins:
words = record_words(prop) - {kind}
owned = {name: field_words(name) for name in record}
scored = [(len(t), name) for name, t in owned.items() if t <= words]
best = max(n for n, _ in scored)
field = only_one(name for n, name in scored if n == best)
left = words - owned[field] - QUALIFIERS - family
if left & other_fields_words or len(left) > len(owned[field] - QUALIFIERS):
raise KeyError("ambiguous or mostly unread") # a crash, never a guess
the readers cover shortcuts (the full shell link: header times and size, link info, string data, tracker and folder blocks), mail, office documents (ooxml core properties, ole2 summary streams and jump lists), event logs, registry hives, packet captures (pcap and pcapng), sqlite databases, plists, iso 9660 and vhd / vhdx disk images, torrents, x.509 certificates and ini style shortcuts. a property no reader answers still falls through to the header guesses and exiftool, and fails loudly after them.
day 15 additions
ooxml relationships. the office document record gained the package's relationships. every .rels part is parsed as xml (attribute order free, entities unescaped: a regex that expected Target before TargetMode and kept & is gone), each Relationship kept with its part, id, type (the last segment of the Type uri), target and whether TargetMode is External. the record's fields: relationship, external and internal targets and types, their counts, hyperlink targets, the attached (remote) template, docProps/custom.xml's named properties and the part list. a property maps to them by its words as before; plurals, rels and references / dependencies are spelled one way on both sides.
for part in z.namelist():
if part.lower().endswith(".rels"):
for el in ElementTree.fromstring(z.read(part)).iter():
if el.tag.split("}")[-1] == "Relationship":
rels.append(Rel(part, el.get("Id"), type_of(el.get("Type")),
el.get("Target"),
(el.get("TargetMode") or "").lower() == "external"))
a list field answers as ranked candidates, each a compact json array: [redacted] first, then the other orders.
day 16 additions
mail threads. day 16's prompt described a mailbox of replies, forwards, duplicates and reconstructed messages, and asked only for one message id, which the mail record (day 14) already held as stored (return to sender). the record now also answers the thread, by the identities rfc 5322 defines: References lists a message's ancestors root first and In-Reply-To names its parent. the reply headers (In-Reply-To, References, Cc, Reply-To, Sender) are always present in the record, empty when the message has none, so a question about an absent one answers the literal none instead of failing.
refs = MSG_ID.findall(record["References"]) # <...> tokens, in order
parent = MSG_ID.findall(record["In-Reply-To"])
own = record.get("Message-ID", "")
record["Thread-Root"] = refs[0] if refs else parent[0] if parent else own
record["Parent-Message-ID"] = parent[0] if parent else refs[-1] if refs else ""
record["Normalized-Subject"] = REPLY_PREFIX.sub("", record.get("Subject", ""))
the subject loses any run of reply and forward prefixes (re:, fwd:, fw:, aw:, sv:, wg:, with an optional [n] count). the record also gains the attachment count and file names (parts with an attachment disposition or a file name), the mime boundary, and the sender's and recipient's bare address and domain. as before, a property maps to these fields by its words.
day 17 additions
media streams by ffprobe's own keys
day 17 asked media-primary-audio-codec-name of a wav (hear me out). the riff header reader had the format tag and the bit depth, not a codec name. a property media-primary-<audio|video|subtitle>-<key> now reads ffprobe's <key> (its underscores written as dashes) of the first stream of that kind, the default-disposition stream first; media-container-<key> reads the format section and media-<kind>-stream-count counts streams. one probe per member, cached; a key the stream lacks is a crash naming it.
stream = primary_stream(data, kind) # ffprobe -show_streams -of json
key = words.replace("-", "_")
if key not in stream:
raise KeyError(f"{prop}: the {kind} stream has no {key!r}")
return str(stream[key])
day 18 additions
composites, near misses and the keys a stream has
day 18 asked media-primary-video-dimensions of an mp4 (frame of reference). dimensions is not a key ffprobe prints; the day 17 reader mapped the words to a key and crashed. a stream or container property now reads, in order: the key as printed (the node's fields, its tags and its side data flattened in); a composite built from the keys the words name; a near miss, every key whose words cover the asked ones, ranked in the node's order; else a crash that names the keys the node has.
def node_value(node, key, composites):
flat = flatten(node) # fields + tags + side data
if key in flat:
return flat[key]
if key in composites:
return composites[key](node) # KeyError names a missing input
words = key.split("_")
near = [k for k in flat if all(w in k.split("_") for w in words)]
if near:
return [flat[k] for k in near] # ranked, stored order
raise KeyError(f"no {key!r} (have {sorted(flat)})")
the stream composites: the stored, coded, display (sample-aspect scaled) and rotated frame, the rotation (side data or the rotate tag), the orientation, the frame rate as a number and as the stored rational, the display aspect ratio, the duration in seconds, the codec, the pixel format, the frame count, whether the coded frame carries padding. the container composites: duration, size, major brand, title, stream count. a composite is a fact, never a verdict: whether one clip is a crop of another is a label no key names. cover art (an attached picture) is never the primary video stream.
video containers in the sniff
ftyp at offset 4 is an mp4 / mov / m4a / 3gp box file; matroska, avi, flv and asf are named by their magic. before day 18 an mp4 read as an unrecognised binary and the triage pointed at binary tools although the media reader had already probed it.
composites of any reader's keys
the same miss can hit every reader that answers an unknown property by its words: a header struct, a record reader, an exiftool tag. each of them now also offers the composites its keys allow, derived from the record itself: a width and a height under one stem give <stem>_dimensions, <stem>_resolution and <stem>_aspect_ratio (reduced, 16:9), a <stem>_ms gives <stem>_seconds. the record's own keys are never overridden and a tie between two composites (an mp4's exiftool ImageDimensions and SourceImageDimensions) is the readers' usual refusal, naming both, never a guess.
for (a, b), names in PAIRS.items(): # ("width", "height") -> dimensions ...
if last == a and (*stem, b) in by_words:
x, y = int(value), int(partner)
out.setdefault(name(stem, "dimensions"), f"{x}x{y}")
out.setdefault(name(stem, "aspect_ratio"), f"{Fraction(x, y).numerator}:{Fraction(x, y).denominator}")
day 19 additions
a header field named without a format word
day 19 asked an elf header field by its struct path, width and format (little endian on the prairie), which the struct table already answered. a name that gives only the field (elf-e-phnum, elf-ei-data) had no reading and failed with an empty list of what the struct has. once every other reader has missed, such a name resolves its words to exactly one struct field and answers with both renderings as ranked candidates; a name the header readers already answer keeps its plain value.
def bare_struct_value(prop, data):
hits = struct_hits(prop.split("-"))
if len(hits) != 1:
return None
return [struct_value(prop + "-dec", data), struct_value(prop + "-hex", data)]
day 20 additions
a field over every record: set values
day 20 asked a mach-o for its cpu type set (universal appeal). the struct table read one field of one record; a trailing set (or list, values, all) word now reads the field over every record of the struct these bytes carry and renders the values as a compact json array: a thin header is one record, a universal binary's fat_arch entries are one record each (20 bytes after the 8-byte fat header, 32 under the 64-bit fat magic). the element rendering and the order are the grader's ([redacted]); the plausible orders go out as ranked candidates.
the bytes pick the struct
two structs of the macho family own a cputype (the thin header and fat_arch), and a thin file names both. a name that fits several structs used to crash as a tie; now, when bytes are at hand, the structs that cannot parse them drop out (a fat reader raises on a thin magic), and a single survivor answers. two survivors are still a crash, never a guess.
day 21 additions
an android package as a record
the metadata category met its first apk on day 21 (package deal). the apk joins the record families (shortcuts, mail, office documents): its record is the manifest's attributes, the launcher activity, the component, permission and deep-link counts and lists, and the first signer, read by the android toolkit. a property resolves against it like any other record, by its words: apk-package-name names package_name, apk-version-name names version_name, and a word that names no field (or names two) is a crash, never a guess. lists answer as ranked compact json arrays, like the office relationship targets.
day 22 additions
the jar record family
day 22 asked a java archive's main class (main character energy). a jar manifest's main section is Key: value lines; a line that starts with one space continues the value before it (the specification wraps at 72 bytes), and the first blank line ends the section, before the per-entry Name: sections of a signed archive. every main attribute is now a record field under its own name (Created-By as created_by), beside the number of class files, so the family's next property is read by its words like any other record.
key, sep, value = line.partition(":")
if line.startswith(" ") and last:
out[last] += line[1:]
elif sep:
last = key.strip()
out[last] = value.strip()
day 23 additions
a field id splits at its stem first
day 23's first sqlite property was d23-sqlite-001-sqlite-page-size-bytes (page me maybe). the property splitter cut ids at the longest known property suffix, and size-bytes (a member's length) is one, so it answered the file size of a member it called d23-sqlite-001-sqlite-page. an id that starts with a dNN-<family>-NNN stem (with or without a mastery setK. or slotK_ prefix) now splits there, and the whole rest is the property; ids without such a stem keep the suffix rule.
FIELD_STEM = re.compile(r"((?:set\d+\.)?(?:slot\d+_)?d\d\d-[a-z0-9]+-\d+)-(.+)")
the sqlite record, every key by its words
the sqlite record (day 14) now carries every header field a property could name, and the page geometry the day's prompt asks to validate:
page = struct.unpack_from(">H", data, 16)[0]
fields = struct.unpack_from(">" + "I" * 14, data, 24)
page_size = 1 << 16 if page == 1 else page
record = {
"page_size": page_size,
"write_version": data[18], "read_version": data[19],
"reserved_bytes": data[20],
"file_change_counter": fields[0], "page_count": fields[1],
"database_size_bytes": page_size * fields[1],
"truncated": "true" if len(data) < page_size * fields[1] else "false",
"freelist_count": fields[3], "schema_format": fields[5],
"text_encoding": fields[8], "user_version": fields[9],
"application_id": struct.unpack_from(">I", data, 68)[0],
"sqlite_version_number": struct.unpack_from(">I", data, 96)[0],
}
the application id is also offered as four ascii characters when it spells them, the text encoding as its name before its number, and the schema's table names and per-table row counts come from sqlite itself, opened read-only and told to tolerate a copy cut short. a sqlite record is read before any generic header guess.
a database embedded past offset 0
the prompt also said "embedded" and "mislabeled". a sqlite-* or pdf-* property on a member whose bytes carry that family's magic later than offset 0 (a database appended to an image, a blob with the wrong extension) is now read from the magic onward; before, the format sniffer saw only the wrapper and the record never ran.
EMBEDDED_MAGIC = {"sqlite": b"SQLite format 3\x00", "pdf": b"%PDF-"}
at = data.find(EMBEDDED_MAGIC[family])
data = data[at:] if at > 0 else data
member-name lists in two orders
zip-central-directory-member-names and its local-header twin answer a json list of the archive's names. until day 23 the archives were written in name order, so directory order and sorted order agreed; day 23's were shuffled, so they disagree. both properties now answer the two orders as ranked candidates, in the order a graded run settled ([redacted]).
day 24 additions
a member named by the leading tokens of its stem
the member resolver maps a <stem>-<property> field to one member of a set: the question's artifact (evidence.zip -> <member>), then a member whose stem is exactly the field's stem, then the stem's trailing number as a whole token, then the set's only member. day 24's roster-filename asked "what is the exact filename of the roster?" with the bundle alone as its artifact: a question whose answer is the name cannot name the member (event horizon). the member was roster- plus a hex suffix, so the exact stem missed and the run raised.
after the exact stem, the resolver now takes the member whose leading name tokens (split at - and _, extension dropped) are the stem's tokens. one such member wins; several are told apart by an ordinal word in the question ("the first evidence note", "the last"), counted over the candidates in sorted order; without an ordinal the older rules run on.
ORDINALS = ("first", "second", "third", "fourth", "fifth",
"sixth", "seventh", "eighth", "ninth", "tenth")
def ordinal(question):
words = set(re.findall(r"[a-z]+", question.lower()))
if "last" in words:
return -1
return next((i for i, w in enumerate(ORDINALS) if w in words), None)
led = sorted((n, b) for n, b in inner.items() if led_by(n, stem))
if len(led) == 1:
return led[0]
nth = ordinal(question_text)
if led and nth is not None and -len(led) <= nth < len(led):
return led[nth]
a member that is exactly the stem still wins first, so older bundles resolve as before.
the event-log record, read as a whole log
the event-log record (day 14) read the file header (chunk numbers, the next record identifier and the count it implies, versions, the chunk count) and the first and last event's system fields. day 24's prompt ("duplicated, truncated ... find the selectively shortened timeline") asked only the record count, but it describes questions about the whole log, so the record now also carries, each answerable by its words:
- the distinct event ids, their count and the most common one;
- the computers, providers and channels seen, and their counts;
- the earliest and latest event times and the span between them in seconds;
- the lowest and highest record ids and the ids missing between them;
- duplicate and distinct records, comparing each record's whole xml rather than its id;
- whether the file is truncated: the header promises
4096 + chunks * 65536bytes, and a shorter file is cut short. this one needs no event parse, so it still answers when the walk fails on a damaged chunk.
"truncated": int(len(data) < 4096 + chunks * 65536),
"expected_size_bytes": 4096 + chunks * 65536,
...
bodies = collections.Counter(event["_xml"] for event in events)
out["duplicate_record_count"] = sum(n - 1 for n in bodies.values() if n > 1)
out["distinct_record_count"] = len(bodies)
day 25 additions
the hive record's whole-tree keys
day 25 asked the first registry hive property, the root key's name (hive mind), which the hive record had read since day 14. the prompt ("renamed and separated from host labels ... regroup the systems, and isolate the planted hive") describes questions about the whole hive, so the record now also carries, each answerable by its words:
root_value_count, the value count at offset 36 of the root key node;key_countandvalue_count, every key and value in the tree, from one walk of the hive;root_value_names, the root key's value names in the order the key lists them, and each root value's data by name, asroot_value_<name>and, when no other key of the record has that name, as bare<name>;checksumandchecksum_valid: the base block stores at 0x1fc the xor of the 127 little-endian dwords before it, with a result of 0 written as 1 and 0xffffffff as 0xfffffffe. a hive edited by hand and not resealed fails the check.
stored = struct.unpack_from("<I", data, 0x1FC)[0]
calc = 0
for (word,) in struct.iter_unpack("<I", data[:0x1FC]):
calc ^= word
calc = {0: 1, 0xFFFFFFFF: 0xFFFFFFFE}.get(calc, calc)
out["checksum"] = stored
out["checksum_valid"] = int(stored == calc)
a hive the tree walker cannot parse keeps its header keys and drops only the tree's.
one word over: a property's synonym
the record guesser maps a property to the record field whose every word the property names, most words first (day 14). day 25's registry-hive-root-name named fewer words than our field: with the family words (registry, hive) set aside, the property's own words are root and name, and the field is root_key_name. no field fit inside the property, and the run failed.
when no field fits, or the best fit leaves a word that another field owns, the guesser now looks for a synonym: a field that holds every own word of the property and at most one word more, qualifiers (path, hex, value and the like) not counted. the property needs at least two own words, the field with the fewest extra words wins, and a tie raises. an extra word that names a thing the record lists (a table of a database record, a value of the hive's root key) picks one thing among several and is never a spelling, so such a field is no synonym: a whole-database row count asked of a database without one does not quietly read one table's.
def synonym(words, family, owned, record):
own = words - family - QUALIFIERS
if len(own) < 2:
return None
named = {w for v in record.values() if isinstance(v, list)
for item in v for w in record_words(str(item))}
near = []
for name, toks in owned.items():
extra = toks - words - QUALIFIERS
if own <= toks and len(extra) <= 1 and not extra & named:
near.append((len(extra), name))
if not near:
return None
fewest = min(n for n, _ in near)
top = sorted(name for n, name in near if n == fewest)
if len(top) > 1:
raise KeyError(f"matches {top}") # a tie fails, never a pick
return top[0]
the property with a leftover word benefits too: a property naming the root key's marker value would match the bare value key and leave root unread, and root_value_<name> holds both words, so it answers instead. the reader inventory has the same tier after its exact matches.
day 26 additions
compressed prefetch: lzxpress huffman in python
day 26's new family was the windows prefetch file (before you go). the prefetch reader had read the uncompressed SCCA format since day 5 and raised on windows 10's compressed form with advice to decompress it on a windows machine. windows 10 and later write prefetch files compressed by default: MAM, a format byte (0x04 for lzxpress huffman, with the high bit set when a crc32 follows), the decompressed size as a u32, then the stream. the reader now decompresses first, in python:
def decompress_prefetch(data: bytes) -> tuple[bytes, bool]:
if data[:3] != b"MAM":
return data, False
size = int.from_bytes(data[4:8], "little")
start = 12 if data[3] & 0x80 else 8
return xpress_huffman(data[start:], size), True
xpress_huffman implements ms-xca's lzxpress huffman. every 64 kib of output starts a block with 256 bytes of four-bit code lengths for 512 symbols, low nibble first; the lengths are expanded into a canonical decode table of 2^15 entries, so one lookup on the top fifteen bits of a 32-bit window yields the symbol and the window shifts by its length. the window refills sixteen bits at a time from little-endian words.
symbol = table[bits >> 17]
bits, extra = (bits << lengths[symbol]) & 0xFFFFFFFF, extra - lengths[symbol]
if extra < 0: # refill from the next u16
bits |= u16(data, pos) << -extra
extra, pos = extra + 16, pos + 2
if symbol < 256:
out.append(symbol) # a literal
else:
length, offset_bits = (symbol - 256) & 0x0F, (symbol - 256) >> 4
if length == 15: # extra length bytes, read from
length = data[pos] + 15 # the byte position, not the bits
pos += 1
if length == 270:
length, pos = u16(data, pos), pos + 2
offset = (bits >> (32 - offset_bits)) + (1 << offset_bits)
... # consume offset_bits, refill
start = len(out) - offset
for i in range(length + 3):
out.append(out[start + i]) # byte by byte: overlaps repeat
a code-length table that does not fill the decode table exactly, and a match that reaches before the start of the output, raise. the decoder needs no windows api and no external tool, and an uncompressed file passes through untouched.
the prefetch record
the record is the SCCA header and its file information block, for format versions 17 (xp), 23 (vista and 7), 26 (8.1) and 30 (10 and 11). each key answers a property by its words, like every other record:
executable_name(andexecutable): sixty bytes of utf-16le at 0x10, cut at the first null;hash: the u32 at 0x4c as eight uppercase hex digits;versionandfile_size(0x00 and 0x0c);compressed, whether the file came in aMAMcontainer.last_run_timeandlast_run_times: the filetimes at 0x80 (0x78 in version 17; one slot before version 26, eight from it), as utc iso times in stored order, which windows writes most recent first.filenamesandfile_count: the null-separated utf-16 strings at the offset and size the file information block gives (0x64, 0x68).volume_count,volume_path(alsovolume_device_path),volume_serialandvolume_created: the first volume entry's device path, serial number as eight hex digits and creation filetime, from the volume information at the offset in 0x6c;directoriesanddirectory_countover every volume, each directory string a u16 length, the characters and a null.
info = struct.unpack_from("<9I", data, 0x54)
_, _, _, _, names_at, names_size, volumes_at, volume_count, _ = info
names = [n for n in data[names_at : names_at + names_size]
.decode("utf-16-le", "replace").split("\0") if n]
path_at, path_len, created, serial, _, _, dirs_at, dir_count = struct.unpack_from(
"<IIQIIIII", data, volumes_at + k * entry_size) # 0x60 bytes in version 30
run_count sits at a version-dependent offset; version 30 comes in two layouts, the 224-byte file information with the metrics array at 0x134 and a layout-specific count offset, and a shorter one with the metrics array at a different offset and the count at 0xc8. the record picks the layout from where the metrics array starts. the first reading subtracted a constant from that offset and landed on a zero dword at 0xc4 on the shorter layout; the review of this page caught it, and the reader now reads 0xc8 there (the day 26 files carry eight run times and a small run count that varies from file to file).
day 28 additions
a disk record for partitioned images
day 28 opened a disk family with a raw image and one question about its partition table (where it all began). an image is a disk when a partition table header (EFI PART) sits at sector 1 for one of the common logical sector sizes, or when its first sector is a partitioned mbr: the boot signature, used entries with a valid status byte and a start past sector 0, and no jump instruction at byte 0 (that would be a volume boot record, which carries the same signature).
the record holds the mbr's columns (signatures, partition types, starts and sector counts, whether it is protective), and for a guid partition table the header (revision, size, crc32 and whether it verifies, its own and the backup header's lba, the usable range, the disk guid, the entry array's lba, count, size and crc32) and the partitions as columns in entry order: start and end lba, size, type guid and the type's common name, unique guid, name and attributes, each sequence as compact json. an mbr-only disk fills the same partition columns from its four primary records. the backup header is checked too (present, valid, equal to the primary), and the image's length against the size the backup header implies, which gives missing-sector, truncation and rebuilt verdicts: the prompt's "prove which image was rebuilt from an incomplete source".
base = head["entry_lba"] * sector
array = data[base : base + head["count"] * head["entry_size"]]
entries = [array[i : i + head["entry_size"]]
for i in range(0, len(array), head["entry_size"])]
used = [(slot, e) for slot, e in enumerate(entries)
if e[:16] != bytes(16)] # zero type guid: unused
parts = [partition(data, e, sector, slot + 1) for slot, e in used]
record["partition_start_lba_sequence"] = compact([p["start_lba"] for p in parts])
a question is answered from the record by its words, as for every other family, so the next disk property is a lookup.
the partition an offset in the question names
the same prompt says "locate evidence by offset". when a question's text names an offset (a byte offset in decimal or hex, a size in kib, mib or gib, or a sector or lba number times the sector size), the record gains evidence_* keys for the partition whose byte range holds it: its index, start, end, size, name and the rest of its columns. the property's "evidence" then means that partition, ahead of a partition that is merely named evidence. an offset in a gap between partitions fails with the layout rather than answering for a neighbour.
filesystem types and container headers
each partition's first sectors give a filesystem type by magic (ntfs, exfat, fat12/16/32 by cluster count, ext2/3/4 by feature flags, xfs, btrfs, hfs+, apfs, iso9660, luks, bitlocker, refs, squashfs, lvm2, swap), and empty when the bytes are all zero, as on day 28's images. the first property every metadata family has asked so far was a header scalar, so a new containers module reads one header record each for the families most likely next: windows crash dumps, hibernation files, ntfs, fat and exfat volume boot records, ext superblocks, qcow2, vmdk (sparse header and descriptor) and ewf. every header scalar is a key by its snake_case words, a volume label is the one bounded read past a header, and a partition that carries one of these headers exposes it under its entry number.
header-first lookups
a property whose words equal exactly one key of a family's header is now answered from the header alone, without parsing the rest of the file (an event log's header, without its records), and an apk's signer fields run the external certificate tool only when one is asked. the metadata mastery's solve halved, with the same answers on every saved fixture.
the winzip-aes key through hashlib
the per-member key of a winzip-aes zip is a pbkdf2-hmac-sha1 derivation. the crypto library we used holds python's interpreter lock while it runs, so the threads that unpack a mastery's sets in parallel ran one at a time; hashlib.pbkdf2_hmac releases the lock. unpacking a script mastery's encrypted members now takes about a third of the time, with byte-identical members.
day 29 additions
six more record readers
on day 29 the sniffer named six kinds no record reader read, so a first question about any of them would have gone unanswered: an $MFT extract, a pdf, a png, a 7z archive, a systemd journal file and a pst mail store. each now has a record of header scalars, names, counts and strings, read in the family's usual way (trust issues read its certificate from the same kind of record):
mft every 1024-byte FILE record (4096 when the extract says so) with
its update-sequence fixups undone; the first record's header
fields on top, every record's file names under file_names, the
record and name counts
pdf the header version, object and page counts, the highest object
number, the info dictionary's strings as stored, the trailer,
and the risky-feature flags (javascript, launch actions,
embedded files, open actions)
png the IHDR fields, the chunk sequence, the text chunks by keyword,
the idat count and size, and every chunk whose crc fails
7z the signature header's version, the start-header crc, the next
header's offset, size and crc, and whether they verify
journal the systemd journal header: signature, compatible and
incompatible flags, state, the file, machine, boot and seqnum
ids, and the u64 fields that follow
pst the format (ansi or unicode by version), the client version,
the encryption kind and the partial crc
the pdf record merges with the existing pdf module's reading so a question answered by either finds it; mft and png join the kinds whose record is tried before the generic header tables.
day 30 additions
the relation family
day 30 asked one field on every tier, d30-cluster-001-relation-identical-sha256-groups, over a zip corpus (seeing double); the lane's readers had zip header fields only and raised. any relation-<words> property over a zip corpus now has a reader, keyed on the family: every non-directory entry is read by its stem, and the words pick the relation (fabricated, provenance, containment, semantic, identical; the first that matches wins). each kind also takes the earlier metadata prompts' own relation words, so renamed copies, clone groups, cropped originals and embedded pairs reach a reader. identical is one form: the entries' sha256 (a digest word picks md5, sha1 or sha512), groups of more than one, stems sorted in each group, groups in first-member order, compact json:
def relation_groups(keyed: list[tuple[str, str]]) -> list[list[str]]:
by: dict[str, list[str]] = {}
for stem, key in keyed:
by.setdefault(key, []).append(stem)
return sorted(sorted(g) for g in by.values() if len(g) > 1)
def identical_groups(entries, digest="sha256"):
return relation_groups(
[(e.stem, hashlib.new(digest, e.data).hexdigest()) for e in entries]
)
the prompt named four more relations ("semantic identity, containment, and claimed provenance", and a fabricated cluster); no tier asked them, so they go out as ranked unsure candidates in the same shape: containment as a container holding another artifact's exact bytes (directly, one archive down, or as a mail attachment), provenance as manifest rows grouped by claimed digest or producer, fabricated as rows the bytes contradict.
semantic keys, near keys and crops
the semantic relation keys a member by what it holds: an image by the sha256 of its decoded rgba pixels, an archive by the sorted digests of its members (names and order dropped), a managed pe by its assembly identity (name, version, culture, public key token; never the mvid), an msi by its UpgradeCode, an apk by its package and signer, audio by its decoded pcm, video by every decoded frame, office and pdf documents by their body text. a later candidate joins near copies: an image by a 64-bit dhash within 10 bits at the same aspect, audio by a tone print, video by its first frame. a crop has another aspect, so it is found by the normalized cross-correlation peak of the crop inside its original at the original's scale, over 0.97 (477 true crops scored at least 0.98, unrelated pairs at most 0.53). a member whose own checksums are stale (a png chunk crc, an inner zip crc: day 30's one-byte decoys) keys by its bytes, never with the group it imitates.
capture hosts and one tool budget
a pcap or pcapng record gains hosts, dns_names, tls_sni and http_hosts from one tshark run, only when the property names a network word (host, dns, sni, tls, http, queries, domains, server, peers, endpoints), with the other spellings as aliases; a packet count runs no tshark. every external tool of one relation read (ffmpeg, ffprobe, 7z, the apk signer's openssl) shares a five-second budget, and past it a member keys by its bytes, so a hung decoder holds a 15-second set for under two seconds instead of ten.
memos keyed by digest
memos keyed on a member's bytes kept every member of a long mastery in memory: 280 one-megabyte images cost 486 MB. media.Kept is a bounded memo whose key is the member's sha256, never its bytes: the image print and pixel key, the exif and ffprobe readings, the audio and video keys, the capture hosts (netcap._HOSTS) and the apk signer's certificates (android._CERTS, which had no memo, so the signer's two openssl runs repeated for every equal apk in every set). a run that raises is never kept. the same 280 images now cost 71 MB, every reading byte-identical.
changelog
- day 1: hashes and sizes of the delivered bytes.
- day 2: magic-prefix properties, member resolution by answer id when questions went stale.
- day 3: byte-order-mark properties, token-aware stem resolution, the format readers above.
- day 4: jpeg start-of-frame dimensions, then header-level width/height for png, gif, bmp and webp with a fallback for unregistered dimension fields.
- day 5: png
IHDRdimensions; a full chunk walk with crc, unknown-chunk, trailing-data, text and geometry checks. - day 6: pdf header version and prepared pdf structure readers; unknown field ids split after the stem's number; an exiftool word-overlap fallback for unknown properties (cached, bounded, never guesses a tie); zip-based formats named by their marker files.
- day 7: raw header-field registry with word matching for unknown properties; zip census properties; nested archives resolvable as members.
- day 8: central-directory and local-header member names; total compressed and uncompressed bytes; guesses must cover every word of the property; family-prefixed generic properties, pdf
startxref, per-type png chunk counts; bounded decompression for every reader. - day 9: header fields by structured name (
<family>-<struct>-<field>-<width>-<format>) over 23 header structs, with width, byte-order and format checks that fail loudly;struct_aliasfor spellings the tables lack; field ids split without a stem number. - day 10: structured names without a struct or a width, kind words checked against the field, closest fields named on a miss; the pe version resource (strings, fixed block, translations, string and fixed versions that must agree); ranked executable labels, data extensions that claim no format and a libmagic-style
cafe babesplit; zip provenance properties. - day 11: pe section, import, export, subsystem, timestamp, debug, checksum, overlay and signature properties, and name comparisons; a generic suffix only after a family word; winzip-aes archives read in memory; a member's zip index parsed once for every reader.
- day 12: a property no table names is tried against the other lanes' readers by exact name before any header or exiftool guess; .net module and assembly columns, attributes and references by name.
- day 13: windows installer databases (the property table and the summary stream, by property words); fields computed across several named members (digests, sizes or any property, ordered or the one at either end).
- day 14: one flat record per artifact for shortcuts, mail, office and ole2 documents, jump lists, event logs, hives, captures, sqlite, plists, disk images, torrents, certificates and ini shortcuts; a property mapped to a record field by its words, refused when the words do not add up.
- day 15: ooxml relationships in the office record, parsed as xml; list fields as ranked compact json.
- day 16: mail thread fields: reply headers (none when absent), thread root and parent by message id, the subject without reply prefixes, attachments, the mime boundary, sender and recipient address and domain.
- day 17: media stream and container properties by ffprobe's keys.
- day 18: media composites (frame sizes, rotation, orientation, frame rate, aspect ratio, duration, codec, pixel format, padding; container duration, size, brand, title), tags and side data by name, near-miss ranking, the have-list in a miss; mp4, matroska, avi, flv and asf sniffed; composites (dimensions, resolution, aspect ratio, seconds) of any header, record or exiftool key pair.
- day 19: a struct field named without a width or format word, answered as ranked renderings once every other reader missed.
- day 20:
setvalues over a struct's records (a thin header or every fat_arch), the fat_arch struct, a name two structs fit resolved by the one that parses the bytes. - day 21: the apk record family (manifest attributes, counts, lists and the first signer).
- day 22: the jar record family (every main-section manifest attribute, continuation lines joined, the class-file count).
- day 23: field ids split at a
dNN-<family>-NNNstem first; the sqlite record's header keys, page geometry and row counts; sqlite and pdf members embedded past offset 0 read from their magic; member-name lists answered in both orders, ranked. - day 24: a member resolved by the leading tokens of the field's stem, several told apart by an ordinal in the question; whole-log event-log keys (event ids, computers, providers, channels, time span, record id gaps, duplicate records, truncation by the header's size).
- day 25: the hive record's whole-tree keys (root value count, key and value counts, root value names and data by name, the base block checksum and its validity); a property's one-word-over synonym among a record's fields, a tie or an extra word naming a listed thing refused.
- day 26: windows 10 prefetch decompressed in python (the
MAMcontainer and ms-xca lzxpress huffman); the prefetch record's name, hash, run times, file and directory lists and first volume's path, serial and creation time. - day 28: a disk record for guid partition tables and mbr-only images (header, entry array and backup checks, partition columns as sequences, completeness verdicts); the partition an offset in the question names; filesystem type by magic per partition; header records for crash dumps, hibernation files, ntfs, fat, exfat, ext2-4, qcow2, vmdk and ewf; header-first lookups and lazy apk signer reads; the winzip-aes key derived through hashlib.
- day 29: record readers for mft extracts, pdf, png, 7z, systemd journals and pst stores (header scalars, names, counts and strings: the kinds the sniffer named but no record read).
- day 30: the relation family over a zip corpus (identical groups in one form; semantic, containment, provenance and fabricated relations as ranked candidates), semantic and near keys per member kind, crops by cross-correlation, a tamper gate, containment through mail attachments; capture hosts, dns names, sni and http hosts on request; one tool budget per relation read; memos keyed by digest.
{generated by a clanker}