seeing double: groups of byte-identical files in a corpus zip

huntress ctf 2026, day 30 · metadata forensics · all three modes

tools used: archive and metadata readers and timed-attempt pipeline.

"the corpus is too large for manual browsing. build a relationship map from exact identity, semantic identity, containment, and claimed provenance, then expose the fabricated cluster." day 30 opened a corpus family for the lane: a zip of small files inside the evidence zip, and one question about it, the groups of files whose sha-256 digests are equal. the prompt named four relations; exact identity was the only one any mode asked for.

the key insight is that the hashing is trivial and the serialization is the whole problem. grouping thirty files by digest is a few lines; what the grader wanted was a particular spelling of each file (the stem, no directory and no extension, which is also the corpus manifest's own id for each file), each group sorted, and the groups ordered by their first member. we had no reader for the family, so the rule was written by hand as six ranked forms, and the grader's form was the third.

challenge shape

single         1 field    the identical-digest groups of cluster-001.zip
time trial    35 fields   the corpus field plus the evidence-note fields,
                          the roster hashes, the cumulative properties of
                          days 4-29 and a synthesis digest order
mastery      490 fields   the same 35 per set, over fourteen sets in one zip

the field was d30-cluster-001-relation-identical-sha256-groups, "determine relation.identical_sha256_groups for cluster-001.zip", artifact metadata-day-30-evidence.zip -> cluster-001.zip, answer kind json: "compact valid json array with exact values and order; no markdown or explanatory text", graded case-insensitively.

artifact anatomy

the single's evidence zip held one member, cluster-001.zip, and that zip held a delivery manifest and thirty files under artifacts/:

delivery-manifest.json        9664 bytes, schema huntress.artifact_delivery_manifest.v1
artifacts/alpha-01.txt ...    six text notes, 57 bytes each
artifacts/bravo-01.ps1 ...    six powershell files, 55 bytes each
artifacts/charlie-01.json ... six json files, 57 bytes each
artifacts/delta-01.png ...    six png images, 244 bytes each
artifacts/echo-01.zip ...     six inner zips, 173 bytes each

the manifest lists every file with an id (alpha-01), its filename (alpha-01.txt), its path (artifacts/alpha-01.txt), a content type, a family, a producer_id (text_template, duplicate_ref, structured_image, structured_zip), the size and the sha-256; on the single every claimed digest matched the bytes. each family's six files were byte-identical: five groups of six.

the time trial's corpus had no manifest: thirty-eight entries under artifacts/ with extensionless names (artifacts/ember-90, artifacts/quill-83), twenty-nine of them unique and the rest in groups of two, three and four.

extraction method

by hand:

import hashlib, io, json, zipfile
from collections import defaultdict
from pathlib import PurePosixPath

def identical_groups(corpus: bytes) -> str:
    by_digest = defaultdict(list)
    with zipfile.ZipFile(io.BytesIO(corpus)) as z:
        for info in z.infolist():
            if info.is_dir() or not info.filename.startswith("artifacts/"):
                continue
            digest = hashlib.sha256(z.read(info)).hexdigest()
            by_digest[digest].append(PurePosixPath(info.filename).stem)
    groups = sorted(sorted(g) for g in by_digest.values() if len(g) > 1)
    return json.dumps(groups, separators=(",", ":"))

three choices in it are the grader's, not the zip's: the stem as the name, singletons left out, and the order. sorting each group and then sorting the list of groups puts the groups in the order of their first member.

our metadata readers had zip header fields only (the end-of-central-directory counts and offsets, the member names), so the property's words matched nothing and the lane reader raised no zip header field for 'relation-identical-sha256-groups'. the rule above was written by hand during the single, and on day 30 it became a reader of relation-<kind> properties over a corpus zip: the identity rule as the one graded form, and ranked candidates for the three relations the prompt named and no question asked (semantic identity, containment, claimed provenance).

validation and answer formatting

the single's graded value:

[["alpha-01","alpha-02","alpha-03","alpha-04","alpha-05","alpha-06"],["bravo-01","bravo-02","bravo-03","bravo-04","bravo-05","bravo-06"],["charlie-01","charlie-02","charlie-03","charlie-04","charlie-05","charlie-06"],["delta-01","delta-02","delta-03","delta-04","delta-05","delta-06"],["echo-01","echo-02","echo-03","echo-04","echo-05","echo-06"]]

the grader rejected, on the single, the same groups spelled as basenames with their extensions (["alpha-01.txt","alpha-02.txt",...]) and as archive paths (["artifacts/alpha-01.txt",...]). on the time trial it rejected the right names in digest order:

rejected  [["quill-83","summit-92"],["cascade-14","delta-70","harbor-37","quartz-16"],["ember-90","lagoon-78","raven-23"]]
accepted  [["cascade-14","delta-70","harbor-37","quartz-16"],["ember-90","lagoon-78","raven-23"],["quill-83","summit-92"]]

the accepted order is lexical by first member (cascade < ember < quill), not the order in which the groups' first files appear in the zip, which would have put the ember-90 group first. the mastery's set 1 shows the same rule with two groups whose first members share a word: [["birch-86",...],["birch-94",...],["estuary-53",...],["kestrel-78",...]]. the json is compact (no spaces after , or :), as the syntax asks.

failed approaches and pivots

automation across modes

the time trial and the mastery read the corpus field through the same rule and every other field through the existing readers: about 0.09 seconds for the time trial's thirty-five fields and about one second for the mastery's fourteen sets.

lessons

solve times

public scoreboard · day 30 finish: below top 10

  • 0:56.0single#5 of 140 solvers
  • 1:00.8time trial#2 of 136 solvers
  • 1:04.4mastery#2 of 136 solvers
  • #14team day finishof 52 teams that cleared it, in 2h43m
  • #5single orderof the day's 7, by our flag time

from the public scoreboard: 09:00 et release to our flag; the table below counts from our attempt's start.

from the mode's first attempt start to its flag (server times). attempts counts our solver runs; wrong submits counts incorrect responses.

mode work to flag attempts wrong submits
single 54.4s 1 2
time trial 2.8s 1 4
mastery 1.9s 1 0

the clock

server timer of the winning attempt; used is attempt start to flag, left is what remained on the clock.

mode time available time used time left
time trial 50.0s 2.8s 47.2s
mastery 445.0s 1.9s 443.1s

where we got stuck

flags: single, time trial, and mastery [redacted].

{generated by a clanker}