seeing double: groups of byte-identical files in a corpus zip
huntress ctf 2026, day 30 · metadata forensics · all three modes
tools used: archive and metadata readers and timed-attempt pipeline.
"the corpus is too large for manual browsing. build a relationship map from exact identity, semantic identity, containment, and claimed provenance, then expose the fabricated cluster." day 30 opened a corpus family for the lane: a zip of small files inside the evidence zip, and one question about it, the groups of files whose sha-256 digests are equal. the prompt named four relations; exact identity was the only one any mode asked for.
the key insight is that the hashing is trivial and the serialization is the whole problem. grouping thirty files by digest is a few lines; what the grader wanted was a particular spelling of each file (the stem, no directory and no extension, which is also the corpus manifest's own id for each file), each group sorted, and the groups ordered by their first member. we had no reader for the family, so the rule was written by hand as six ranked forms, and the grader's form was the third.
challenge shape
single 1 field the identical-digest groups of cluster-001.zip
time trial 35 fields the corpus field plus the evidence-note fields,
the roster hashes, the cumulative properties of
days 4-29 and a synthesis digest order
mastery 490 fields the same 35 per set, over fourteen sets in one zip
the field was d30-cluster-001-relation-identical-sha256-groups, "determine relation.identical_sha256_groups for cluster-001.zip", artifact metadata-day-30-evidence.zip -> cluster-001.zip, answer kind json: "compact valid json array with exact values and order; no markdown or explanatory text", graded case-insensitively.
artifact anatomy
the single's evidence zip held one member, cluster-001.zip, and that zip held a delivery manifest and thirty files under artifacts/:
delivery-manifest.json 9664 bytes, schema huntress.artifact_delivery_manifest.v1
artifacts/alpha-01.txt ... six text notes, 57 bytes each
artifacts/bravo-01.ps1 ... six powershell files, 55 bytes each
artifacts/charlie-01.json ... six json files, 57 bytes each
artifacts/delta-01.png ... six png images, 244 bytes each
artifacts/echo-01.zip ... six inner zips, 173 bytes each
the manifest lists every file with an id (alpha-01), its filename (alpha-01.txt), its path (artifacts/alpha-01.txt), a content type, a family, a producer_id (text_template, duplicate_ref, structured_image, structured_zip), the size and the sha-256; on the single every claimed digest matched the bytes. each family's six files were byte-identical: five groups of six.
the time trial's corpus had no manifest: thirty-eight entries under artifacts/ with extensionless names (artifacts/ember-90, artifacts/quill-83), twenty-nine of them unique and the rest in groups of two, three and four.
extraction method
by hand:
import hashlib, io, json, zipfile
from collections import defaultdict
from pathlib import PurePosixPath
def identical_groups(corpus: bytes) -> str:
by_digest = defaultdict(list)
with zipfile.ZipFile(io.BytesIO(corpus)) as z:
for info in z.infolist():
if info.is_dir() or not info.filename.startswith("artifacts/"):
continue
digest = hashlib.sha256(z.read(info)).hexdigest()
by_digest[digest].append(PurePosixPath(info.filename).stem)
groups = sorted(sorted(g) for g in by_digest.values() if len(g) > 1)
return json.dumps(groups, separators=(",", ":"))
three choices in it are the grader's, not the zip's: the stem as the name, singletons left out, and the order. sorting each group and then sorting the list of groups puts the groups in the order of their first member.
our metadata readers had zip header fields only (the end-of-central-directory counts and offsets, the member names), so the property's words matched nothing and the lane reader raised no zip header field for 'relation-identical-sha256-groups'. the rule above was written by hand during the single, and on day 30 it became a reader of relation-<kind> properties over a corpus zip: the identity rule as the one graded form, and ranked candidates for the three relations the prompt named and no question asked (semantic identity, containment, claimed provenance).
validation and answer formatting
the single's graded value:
[["alpha-01","alpha-02","alpha-03","alpha-04","alpha-05","alpha-06"],["bravo-01","bravo-02","bravo-03","bravo-04","bravo-05","bravo-06"],["charlie-01","charlie-02","charlie-03","charlie-04","charlie-05","charlie-06"],["delta-01","delta-02","delta-03","delta-04","delta-05","delta-06"],["echo-01","echo-02","echo-03","echo-04","echo-05","echo-06"]]
the grader rejected, on the single, the same groups spelled as basenames with their extensions (["alpha-01.txt","alpha-02.txt",...]) and as archive paths (["artifacts/alpha-01.txt",...]). on the time trial it rejected the right names in digest order:
rejected [["quill-83","summit-92"],["cascade-14","delta-70","harbor-37","quartz-16"],["ember-90","lagoon-78","raven-23"]]
accepted [["cascade-14","delta-70","harbor-37","quartz-16"],["ember-90","lagoon-78","raven-23"],["quill-83","summit-92"]]
the accepted order is lexical by first member (cascade < ember < quill), not the order in which the groups' first files appear in the zip, which would have put the ember-90 group first. the mastery's set 1 shows the same rule with two groups whose first members share a word: [["birch-86",...],["birch-94",...],["estuary-53",...],["kestrel-78",...]]. the json is compact (no spaces after , or :), as the syntax asks.
failed approaches and pivots
- basenames with their extensions and the archive paths: both rejected on the single. the manifest's
idcolumn already spelled the accepted form; reading the manifest first would have put the stem first. - groups ordered by digest: rejected on the time trial. a digest order is stable but arbitrary to a reader; the grader sorts by name.
automation across modes
the time trial and the mastery read the corpus field through the same rule and every other field through the existing readers: about 0.09 seconds for the time trial's thirty-five fields and about one second for the mastery's fourteen sets.
lessons
- for a "groups" answer, the grouping is the easy part; settle the member spelling, the singleton rule and both orders before the first submit.
- a corpus manifest names its files; its
idcolumn is the spelling a question about the corpus most likely uses. - when a list of candidate forms is reused on new bytes, its positions can change meaning: here three name forms collapsed into one on extensionless names, and the form that had won the single was no longer at the position that had won it.
solve times
public scoreboard · day 30 finish: below top 10
- 0:56.0single#5 of 140 solvers
- 1:00.8time trial#2 of 136 solvers
- 1:04.4mastery#2 of 136 solvers
- #14team day finishof 52 teams that cleared it, in 2h43m
- #5single orderof the day's 7, by our flag time
from the public scoreboard: 09:00 et release to our flag; the table below counts from our attempt's start.
from the mode's first attempt start to its flag (server times). attempts counts our solver runs; wrong submits counts incorrect responses.
| mode | work to flag | attempts | wrong submits |
|---|---|---|---|
| single | 54.4s | 1 | 2 |
| time trial | 2.8s | 1 | 4 |
| mastery | 1.9s | 1 | 0 |
the clock
server timer of the winning attempt; used is attempt start to flag, left is what remained on the clock.
| mode | time available | time used | time left |
|---|---|---|---|
| time trial | 50.0s | 2.8s | 47.2s |
| mastery | 445.0s | 1.9s | 443.1s |
where we got stuck
- single, about 52 seconds: the lane reader had no relation reader and crashed within two seconds of the attempt's start. the lane fixer's first command came about ten seconds later, and it spent about forty seconds composing a handler, with nothing saved meanwhile. the handler returned six ranked forms, three name spellings (basename, archive path, stem) in two group orders (by first member, by digest), and the grader took the third after rejecting the first two, 54.4 seconds after the attempt began. the manifest's
idcolumn would have ranked the stem first. - time trial, about two seconds: the corpus names had no extension, so the basename and stem spellings were the same string and the six forms became four. the time trial led with the form at the single's winning position, which was now a by-digest order; the grader rejected it, and the next form, the correct one, went inside the same attempt. the four incorrect responses all carried that one wrong form.
- the three other relations the prompt named never became questions on day 30.
flags: single, time trial, and mastery [redacted].
{generated by a clanker}