sha1-hulud: reversing a self-modifying flag validator

huntress ctf 2026, day 30 · binary reverse engineering · static challenge, single mode

the day-30 binary was a single windows x64 executable. it prints What is the flag?, reads a line, and answers either with silence or with correct, the flag is followed by whatever you typed. behind that prompt sits a rust validator that combines four sha-1 variants, self-modifying code with INT3 and vectored exception dispatch, two 45-million-iteration mixing loops, environment fingerprints, a nested payload decoder, and a two-pass custom block cipher before a single byte is ever compared.

the key insight is that a validator you can execute does not need to be broken — it can be made to tell you everything. one instrumented run of the real code reveals both the comparison's target ciphertext and every cipher key in play, and both cipher passes are invertible, nonlinear map included: x + (x*x | 5) falls to a bit-by-bit reconstruction from its least significant bit. the one thing that cannot be inverted offline is the first pass's key schedule, because its keys depend on the plaintext it has just encrypted. the working trick is to snapshot the emulator and replay the executable one recovered block at a time, letting the original instructions derive each next key.

challenge shape

download: sha1-hulud.exe, 632,320 bytes
prompt:   What is the flag?
success:  correct, the flag is <input>
flag:     flag{[0-9a-f]{32}}
modes:    one static challenge, no timed tiers

the challenge page describes an md5-shaped flag, but the validator's plaintext buffer carries the full flag{...} wrapper: a length-prefixed 38-byte input. there is no md5 preimage attack anywhere in this solution. the recovery is a complete white-box extraction of the target ciphertext and rolling keys, followed by ordinary cipher inversion.

artifact anatomy

property value
file sha1-hulud.exe
size 632,320 bytes
sha-256 7538868046a98ad42b424df2b4f8652319d22b388ebf577ba224e3ca73475f8b
architecture pe32+, x86-64
image base 0x140000000
entry point 0x14007a5c4
main validator 0x14003c38a

every constant in this post belongs to that exact file. a second download of the same challenge had a different hash, different decoder parameters and a different embedded body — nothing was interchangeable. match the hash, then re-derive the numbers on yours.

section rva virtual size raw size
.text 0x1000 0x7d976 0x7da00
.rdata 0x7f000 0x18994 0x18a00
.data 0x98000 0x1d84 0xc00
.pdata 0x9a000 0x1ed8 0x2000
.gfids 0x9c000 0x80 0x200
.tls 0x9d000 0x21 0x200
.xsobf 0x9e000 0x1ad 0x200
_RDATA 0x9f000 0x1f4 0x200
.reloc 0xa0000 0x864 0xa00

the main validator is about 122 kb of code. whole-function decompilation is unwieldy; disassembly, small helper decompilations, cross-references and instrumented execution together carry the analysis.

four large routines at 0x140001080, 0x140010840, 0x14001dfc0 and 0x14002b840 implement sha-1-like compression. the first matches ordinary sha-1's 80 working-variable rounds. the other three do not — replacing all four with a standard sha-1 produces plausible-looking intermediate data and a wrong answer. each entry point receives an 88-byte context:

offset contents
0x00 five little-endian 32-bit working-state words
0x14 round/index field
0x18 sixteen little-endian 32-bit message words

the caller has already converted message bytes into the word representation; the routines return the five working variables and the caller handles feed-forward addition. treating the routines as complete digest functions adds the state at the wrong layer.

the data the validator leans on:

address contents
rva 0x87e80 first-pass rotation table, 128 bytes, two little-endian u32 entries per round
rva 0x87f28 second-pass rotation table, same layout
rva 0x7fa50 16 kb embedded payload, header 88aef9ad…

and the validator internals that mattered:

address purpose
0x14003c33b environment probe-vector gate
0x14003cee4 normalized .text hash checkpoint
0x14003d1fe first decoder-parameter checkpoint
0x14004194d, 0x1400481db expensive mixing-loop entries
0x140058548 first-pass block input and current key
0x14005868d first-pass block output
0x1400586e0 complete first-pass ciphertext
0x1400589cd complete second-pass ciphertext and key
0x140058b0f original ciphertext byte before masking
0x140059cd9 final comparison xors
0x14005a02b successful control-flow path

extraction method

read the strings first

several strings are pairs of equally sized byte arrays; xor recovers them:

from pathlib import Path
import pefile

data = Path("sha1-hulud.exe").read_bytes()
pe = pefile.PE(data=data)
base = pe.OPTIONAL_HEADER.ImageBase

def xor_literal(first_va, second_va, length):
    left = pe.get_data(first_va - base, length)
    right = pe.get_data(second_va - base, length)
    return bytes(a ^ b for a, b in zip(left, right))

print(xor_literal(0x1400883A8, 0x1400883B9, 17))
print(xor_literal(0x1400883CE, 0x1400883E3, 21))
b'What is the flag?'
b'correct, the flag is '

the 429-byte .xsobf section is a counter-based xor stream:

MASK64 = (1 << 64) - 1

def rol64(value, count):
    count %= 64
    return ((value << count) | (value >> (64 - count))) & MASK64

def xsobf_decode(blob, offset, length, tag):
    out = bytearray()
    counter = 0
    for i in range(length):
        value = counter ^ rol64(tag, 17) ^ 0x5D2E9C47A1B386F0
        value = ((value ^ (value >> 29)) * 0x41169DDC6C156DB1) & MASK64
        value = ((value ^ (value >> 27)) * 0xA3D1F5BB9847A411) & MASK64
        out.append(blob[offset + i] ^ ((value >> 55) & 255)
                   ^ ((value >> 24) & 255))
        counter = (counter + 0x65767AC2BA816C3) & MASK64
    return bytes(out)

blob = pe.get_data(0x9E000, 429)
for offset, length, tag in [
    (0, 8, 0xC260001C),
    (8, 33, 0xC250001C),
    (41, 8, 0xC4400021),
    (49, 11, 0xC800000D),
]:
    print(xsobf_decode(blob, offset, length, tag))

this reveals registry paths, a drivers directory fragment and cheatengine. none of it is flag material — the decoded bytes are leads into the anti-analysis checks. trace what consumes a string before believing the loop around it produces flag bytes.

run the real code, not the static bytes

a bare cpu emulator is not enough. the validator expects windows loader structures: a peb and teb, tls initialization, import resolution, standard input, vectored exception handling, and a long list of system queries. map the pe at its preferred base, synthesize the dll exports, and answer the queries coherently (GetSystemInfo, GlobalMemoryStatusEx, RtlGetVersion, firmware tables, FindFirstFileW on the drivers directory, process snapshots — the trace is long). the tls callbacks run before the validator.

an early silent exit comes from a nine-byte probe vector: at 0x14003c33b, any byte equal to 1 terminates. passing that gate is necessary but nowhere near sufficient, because environment-derived values also feed the decoding path.

the environment fingerprints reach the keys

several large, opaque functions look like flag generators and are actually environment fingerprints. their state feeds scalar wrappers that later produce decoder parameters and cipher keys. the folds involved obey an invariant of the form:

s0' + s1' = s0 + s1 + (left - right) * odd_multiplier  (mod 2^64)

when the two observations agree, the difference vanishes and the initialized state sum — the original nonce — survives. an emulator whose answers are inconsistent perturbs the nonce, and nothing fails loudly: the decoder just quietly produces wrong parameters. the final instrumentation restores the original nonce at eight wrapper call sites, reading the executable's own raw nonce table:

call sites register offset in raw nonce table
0x14003dce3, 0x1400444ca rcx 8
0x14003f51c, 0x140045cca r8 24
0x140040bb4, 0x140047403 rcx 32
0x140041199, 0x1400479f3 rcx 16

at these points the raw table pointer is the qword at [rbp+0x1160] (that stack slot is reused elsewhere; the interpretation is specific to these sites):

def unwrap_nonce(self, uc, address, size, _):
    register, offset = NONCE_CALLS[address]
    value = self.emu.u64(self.q(0x1160) + offset)
    self.setreg(register, value)

one family arrived as 0x439e47fe8f9fb43e while its raw nonce was 0x54edcebaa16c4a9f; the other three already matched under the final environment model. a coherent windows model and these corrections were both required — zeroing the probe vector alone establishes nothing downstream.

keep the self-modifying sha-1 routines intact

the four compression routines decrypt their own instruction bytes as they execute, use INT3 with vectored exception handling, and restore their code after each call. static disassembly of the encrypted image misses the effective arithmetic, and an emulator that keeps translated blocks across a self-modification executes stale code.

for the compression path the relevant exception mode sets r15 to the trap address and resumes after the one-byte INT3. an original-code oracle — a second unicorn instance mapping the unmodified pe — supplies that behavior, a thread identity at gs:[0x48], the matching cached thread value, and mode 0xc8c291de at rva 0x98ae0:

def trap(self, uc, number, _):
    if number != 3:
        raise RuntimeError(f"Unexpected compression exception: {number}")
    # Unicorn has already advanced RIP past the INT3 instruction.
    uc.reg_write(X.UC_X86_REG_R15, uc.reg_read(X.UC_X86_REG_RIP) - 1)

def call(self, entry, context):
    self.uc.mem_write(CONTEXT, context)
    self.uc.mem_write(STACK + 0x8000, struct.pack("<Q", RETURN))
    self.uc.reg_write(X.UC_X86_REG_RSP, STACK + 0x8000)
    self.uc.reg_write(X.UC_X86_REG_RCX, CONTEXT)
    self.uc.reg_write(X.UC_X86_REG_RDX, (1 << 64) - 1)
    self.uc.ctl_remove_cache(BASE + 0x1000, BASE + 0x3AB80)
    self.uc.emu_start(entry, RETURN, timeout=10_000_000, count=200_000)
    if self.uc.reg_read(X.UC_X86_REG_RIP) != RETURN:
        raise RuntimeError("Compression routine did not return")
    return bytes(self.uc.mem_read(CONTEXT, 88))

the cache invalidation before every call is the load-bearing detail. we checked the oracle against the original self-modifying routines on three generated contexts per routine: all twelve outputs matched, the untouched context bytes were unchanged, and the original .text bytes were restored after each call.

integrity is checked, so preserve it

the executable hashes a normalized form of its own .text — its treatment of relocations and pe metadata included. hashing the file's raw .text bytes is not equivalent. the successful replay reaches this checkpoint:

1a3fb092010e4509a0d94a11100e2dd1a16148ac

.text raw size (0x7da00) exceeds its virtual size (0x7d976); restore the full section.get_data() into the mapped image before running the validator. truncating at VirtualSize changes what the integrity path sees.

decode the embedded payload

the payload at rva 0x7fa50 starts with the 16-byte header 88aef9ad…, followed by a variable-length integer stream whose continuation convention is backwards from usual leb128: a byte with its high bit clear starts the next integer; high-bit-set bytes extend the current one:

def varints(data):
    position = 0
    while position < len(data):
        value = data[position] & 127
        position += 1
        while position < len(data) and data[position] & 128:
            value = (value << 7) | (data[position] & 127)
            position += 1
        yield value

over the inspected 0x4000-byte region this yields 8,273 integers; at eleven output bits per integer that is a nominal packed capacity of 11,375 bytes. the first decoder's parameters for this artifact were:

(11, 37, 16, 2, 3, 27)

and the decoded stream begins with the four-byte prefix 37 08 00 00 — little-endian 2,103, a decoder-stage length, not the final input length. treat all of these as checkpoints to hit, not as a from-scratch reimplementation: let the executable run its own nested decoder and record what it produces.

evaluate the 45-million-iteration loops exactly

two loops (0x14004194d, 0x1400481db) run 45,000,000 iterations each, and their final state feeds later decoding. skipping them or substituting zero corrupts the solve. their recurrence is:

offset = ((value >> 32) * (len(data) - 7)) >> 32
word = int.from_bytes(data[offset:offset + 8], "little")
value = ((value ^ word) * 0xF8714DF9DBA9A23D) & ((1 << 64) - 1)
value ^= value >> 29

the offset formula always selects a complete eight-byte window while the input has at least eight bytes, so the loops evaluate exactly and quickly outside instruction-by-instruction emulation:

from functools import lru_cache

@lru_cache(maxsize=8)
def long_mix(value, data, iterations, multiplier):
    if len(data) < 8:
        raise ValueError("Mixer input is too short")
    words = [int.from_bytes(data[i:i+8], "little")
             for i in range(len(data) - 7)]
    span = len(data) - 7
    mask = (1 << 64) - 1
    for _ in range(iterations):
        offset = ((value >> 32) * span) >> 32
        value = ((value ^ words[offset]) * multiplier) & mask
        value ^= value >> 29
    return value

at each hook, write back the recurrence result and the terminal loop counter, then resume after the loop. other long loops are fingerprint or decoy paths — follow their consumers before deciding to accelerate, model or ignore them. keep natural outer-loop control flow: forcing an iteration counter changes a downstream salt and invalidates the target.

pull the target out of the comparison

the decisive comparison is an xor accumulator near the end of the validator:

0x140059cd9    xor bl, sil
0x140059cdc    xor bl, dl
0x140059cde    or  byte ptr [rbp+0xca8], bl
...
0x140059ce9    test accumulated result for zero

every compared byte must contribute zero. an earlier mask is applied to the ciphertext byte before it reaches bl, so recording only sil ^ dl is not enough. record the original ciphertext byte O at 0x140058b0f, before its first mask. at 0x140059cd9 let B = bl, S = sil, D = dl. because B already contains O XOR first_mask, the target byte is:

T = O XOR B XOR S XOR D

this cancels the chosen input's ciphertext contribution and keeps the comparison's target and masks:

target = bytes(
    original ^ bl ^ sil ^ dl
    for original, (bl, sil, dl) in zip(original_bytes, compare_rows)
)

one instrumented run with any input yields the full target. this artifact's three-block (48-byte) target:

fd14bd19…
d3902ac9…
68c12755…

invert the block ciphers

the plaintext is built as:

LE64(input length) || normalized input || zero padding to a 16-byte multiple

both passes work on 128-bit blocks as two 64-bit words, sixteen rounds, rotation tables from the executable, a keyed mixer, and the nonlinear map:

P(x) = x + (x*x OR 5)  (mod 2^64)

despite its appearance, P inverts bit by bit from the bottom: once the lower bits of x are known, the next output bit determines the next input bit, because for bit positions above zero, flipping input bit i does not change the square's bits at or below i — the addition contributes the deciding bit. the forced low bits from OR 5 do not obstruct the construction:

def unpoly(target):
    value = 0
    for bit in range(64):
        if ((value + (value * value | 5)) ^ target) & (1 << bit):
            value |= 1 << bit
    assert (value + (value * value | 5)) & ((1 << 64) - 1) == target
    return value

everything else reverses directly: subtraction undoes addition, xor undoes xor, and the left/right xor-shifts invert by repeated substitution. the complete inverse block function:

import struct

M = (1 << 64) - 1

def rol(value, count):
    count %= 64
    return ((value << count) | (value >> (64 - count))) & M

def mix(value):
    value = ((value ^ (value >> 29)) * 0xB89294A129AD1091) & M
    value = ((value ^ (value >> 35)) * 0xF6E2E6BFAB39E8F7) & M
    return value ^ (value >> 26)

def unright(value, count):
    result = value
    for _ in range(64 // count):
        result = value ^ (result >> count)
    return result & M

def unleft(value, count):
    result = value
    for _ in range(64 // count):
        result = value ^ ((result << count) & M)
    return result & M

def unpoly(target):
    value = 0
    for bit in range(64):
        if ((value + (value * value | 5)) ^ target) & (1 << bit):
            value |= 1 << bit
    assert (value + (value * value | 5)) & M == target
    return value

def inv(block, key, rotations, seed, kind):
    a, b = struct.unpack("<QQ", block)
    k0, k1 = struct.unpack("<QQ", key)
    increment = (0x80E29AA772AA64A6 if kind == 1
                 else 0x0633C3366C503E172)
    lane_increment = (0x40714D53B9553253 if kind == 1
                      else 0xB19E19B36281F0B9)

    for i in range(15, -1, -1):
        if kind == 1:
            aa = unright(unleft(b, 21), 10)
            bb = unleft(unright(a, 33), 14)
        else:
            aa = unright(unleft(b, 26), 30)
            bb = unleft(unright(a, 29), 24)

        bb ^= rol(aa, rotations[i * 8 + 4])
        aa ^= rol(bb, rotations[i * 8])
        round_seed = (seed + i * increment) & M
        a = (unpoly(aa) - mix(k0 ^ round_seed)) & M
        b = unpoly(bb) ^ mix(k1 ^ ((round_seed + lane_increment) & M))

    return struct.pack("<QQ", a, b)

def xor(left, right):
    return bytes(a ^ b for a, b in zip(left, right))

the first rotation table is 128 bytes at rva 0x87e80, the second at 0x87f28; each round has two 32-bit rotation entries, and reading the low byte of each entry is enough because the machine rotations use the low six bits of the count.

before trusting any inverse, apply it to a ciphertext generated from a known placeholder input and assert it recovers the observed intermediate data.

undo the reverse pass first

let y_i be the first pass's ciphertext and c_i the second pass's. the second pass walks blocks from the end toward the beginning:

c_i = G_i(y_i XOR c_(i+1))
c_N = 0

so:

y_i = inverse_G_i(c_i) XOR c_(i+1)
def inverse_reverse_pass(ciphertext, key, rotations):
    count = len(ciphertext) // 16
    output = [b""] * count
    chain = bytes(16)

    for iteration, index in enumerate(range(count - 1, -1, -1)):
        block = ciphertext[index * 16:(index + 1) * 16]
        seed = (count * 0x33C3366C503E1720
                + 0x7DDAE3471243D999
                + iteration * 0xCC3CC993AFC1E8E0) & M
        output[index] = xor(inv(block, key, rotations, seed, 2), chain)
        chain = block

    return b"".join(output)

undoing the second pass on the extracted target produced:

5a82c3a7…
46826d0e…
9e34311d…

recover the rolling keys by replaying the first pass

the forward pass chains as:

y_i = F_key_i(p_i XOR y_(i-1))
y_(-1) = 0

so p_i = inverse_F_key_i(y_i) XOR y_(i-1). but the key state evolves with the plaintext: collecting all keys from a run with an arbitrary input and reusing them offline fails as soon as the state diverges. let the executable advance its own state:

  1. start with a placeholder input of 32 zero characters.
  2. before the first forward block, snapshot registers, flags, image, stack and allocated heap.
  3. let the validator reach the real comparison and derive the target ciphertext.
  4. undo the reverse pass with the observed second-pass key.
  5. invert the first forward block with its observed initial key.
  6. check the recovered length prefix against the observed number of blocks:
length = int.from_bytes(first_block[:8], "little")
if length <= 65536 and (length + 8 + 15) // 16 == block_count:
    # structurally consistent; restore the snapshot and recover this case
    ...
  1. restore the snapshot, then recover and inject one block at a time at the forward-pass input; let the original instructions calculate the next key and chaining state after each injected block.

the core recovery hook, at the first-pass block input (rsi is the block index, the key and chaining state live at fixed offsets from rbp):

index = self.reg("rsi")
rbp = self.reg("rbp")
key = self.read(rbp + 0x710, 16)

white = inv(self.target1[index * 16:(index + 1) * 16], key,
            self.rot1, 0x40714D53B9553253, 1)
plain = xor(white, self.read(rbp + 0x878, 16))
self.plain[index * 16:(index + 1) * 16] = plain

# supply the recovered block at the original forward-pass input
uc.mem_write(rbp + 0xCC0, white)
uc.mem_write(rbp + 0xD10, white)
uc.mem_write(rbp + 0x610, plain)

the successful case was outer index 0, inner index 1, with three blocks. its recovered plaintext blocks:

2600000000000000666c61677b[redacted]
[redacted]
[redacted]

the leading 0x26 is 38 decimal, and the following bytes begin flag{ — the length prefix corrected an early assumption that the input was exactly 32 characters.

validation and answer formatting

strip the eight-byte length prefix, take exactly the length it names (38 bytes), and keep the flag{...} wrapper — the last two bytes are block padding, and slicing a fixed 32 bytes after the prefix would truncate the answer:

flag{[redacted]}

validate separately from recovery. the replay above injects plaintext blocks so the key schedule can advance; it is a derivation tool, not proof. build a fresh emulator with the recovered 38-byte answer, keep the same environment model, the same original-code hash helpers, the same exact mixing-loop evaluation and the same nonce normalization, and disable every recovery and comparison observation hook. that replay reaches 0x14005a02b — the validator's own success path:

RECOVERED: flag{[redacted]}
ORIGINAL COMPARE PATH: ACCEPTED

the same fresh replay with 32 zero characters prints nothing and exits without reaching any success branch. the submission of the wrapped form was accepted, which is the external confirmation.

failed approaches and pivots

approach why it failed what worked
force the success branch success echoes the supplied input follow the comparison's data flow
search decoded strings for flag material they belong to environment checks trace their consumers
use standard sha-1 for all four routines three compression functions differ execute their original effective code
trust the initial disassembly code changes during execution observe decrypted execution and invalidate cached translations
pass only the nine-byte gate fingerprints also perturb later nonces preserve coherent state and restore nonce invariants
skip the 45-million-iteration loops their final states feed decoding evaluate the exact recurrence outside instruction emulation
force outer-loop termination the loop index contributes to later state keep natural branch flow
reuse first-pass keys from a placeholder run keys evolve with recovered plaintext restore a snapshot and advance one recovered block at a time
assume a 32-byte plaintext the executable includes the flag{...} wrapper parse the recovered little-endian length
copy another artifact's constants per-download artifacts differ match file hashes and local checkpoints

lessons

solve times

public scoreboard · day 30 finish: below top 10

  • 2h43msingle#14 of 52 solvers
  • #14team day finishof 52 teams that cleared it, in 2h43m

from the public scoreboard: 09:00 et release to our flag; the table below counts from our attempt's start.

from first download to the accepted flag; the nine wrong submits were candidate values tried against the checker before the recovery pipeline existed.

mode work to flag attempts wrong submits
single 2h43m47s 1 9

where we got stuck

flags: single [redacted].

{generated by a clanker}