CoilVM Writeup

Overview

CoilVM is a reverse-engineering challenge built around a nanomite crackme: a stripped x86-64 ELF that reads a password from stdin and, if it's correct, unseals and prints a flag. The comparison routine itself contains no useful logic at all, just a wall of int3 software breakpoints. The real per-byte check lives inside the SIGTRAP handler that catches each trap, and that handler re-keys itself with a running FNV-1a hash after every accepted byte. To recover the flag you have to recognize the nanomite pattern, reconstruct the handler's key schedule, and emulate it forward to recover a 36-byte password. There is no flat system of equations a solver like Z3 can consume in one shot.

Reconnaissance

main() does four things worth noting before you touch the comparison function:

1. mprotects the page containing check_coil to RWX, a hint that something in that region gets written at runtime.
2. Installs an SA_SIGINFO handler for SIGTRAP.
3. Calls register_sites(), then reads exactly 36 bytes from stdin.
4. Calls check_coil() and, only if nothing failed, calls unseal_and_print().

Running strings on the binary turns up no SkillBit{...} anywhere, but it does turn up a decoy:

C01L_VM_d3c0y_d0_n0t_b3l13v3_th15_str1ng


The binary is stripped, so there are no function names to read, but the string is easy to follow: one short routine loads it and strcmps the input against it. That routine has the shape of a crackme's password check, and nothing on main()'s accepting path ever calls it. Typing the string in gets you nope, same as any other wrong guess. It is bait for a quick strings | strcmp guess.

Exploitation

Recognizing the nanomites

Disassembling check_coil directly is useless. It is a long run of 0xCC bytes interleaved with loads from a scratch variable. Each 0xCC traps into the kernel and delivers SIGTRAP. Because int3 is a trap (not a fault), the saved RIP in the signal's ucontext already points one byte past the breakpoint, so the handler can just return and execution resumes cleanly at the next site, with no RIP fix-up needed. That is what lets the whole comparison hide behind code that disassembles into nothing.

The RWX mapping from main() explains itself here. Each trap rewrites one scratch byte in the .text page, which looks alarming on first sight (self-modifying code!), but the rewritten byte is never executed as an instruction. It desyncs a naive linear disassembly sweep and nothing more, so chasing it leads nowhere.

register_sites() runs check_coil() once in a calibration mode before ever reading input, so the handler can record the resume-RIP of each trap in the order they fire. That builds a lookup table mapping RIP to a byte index, the nanomite table, which the real (non-calibration) handler invocation uses to know which position in the password it's currently checking.

Reading the handler as the real VM

The comparison logic, reconstructed from the handler, is this per-trap model:

state = 0x811C9DC5                               # FNV-1a offset basis
for i in range(36):
   target = NM_ENC[i] ^ (state & 0xFF)
   t      = ROL8(input[i] ^ (state & 0xFF), (state >> 5) & 7)
   accept iff t == target
   state  = (state ^ input[i]) * 0x01000193     # FNV-1a prime, 32-bit wrap


NM_ENC is a 36-byte table sitting in .rodata; 0x811C9DC5 and 0x01000193 are the standard FNV-1a 32-bit basis and prime, and they show up as plain 32-bit immediates in the disassembly once you know to look for them.

Block i's target is XOR-masked by state, and state at block i is a function of every earlier accepted byte. So there is no fixed system of 36 independent equations to hand a solver; each equation resolves only once you have solved the ones before it. The way through is to emulate the schedule forward.

The rdtsc gate

coil_step() (the per-trap handler body) reads rdtsc at the start and end of each invocation. If the gap since the previous trap exceeds a fixed cycle threshold, it XORs state with 0xA5A5A5A5 before continuing. A ptrace single-step loop blows straight through that threshold, whether you are stepping instruction by instruction under gdb or intercepting and re-delivering each SIGTRAP by hand. The run still completes and still prints nope, with nothing to say what went wrong, because the poisoned key makes every later target decode wrong. Under a debugger you also have to be careful that int3 isn't intercepted by the debugger itself before it ever reaches your handler. Reconstructing the schedule offline and running the numbers in Python avoids all of that.

Recovering the password

There is no per-byte oracle to grind against. The binary reports pass or fail for all 36 bytes at once, and a 36-byte printable password is a space of 94^36, far past brute force. The schedule has to be emulated.

That is cheap to do, because the per-byte transform is invertible. Walk the same schedule forward and, at each step, solve for the input byte the handler would have accepted:

shift  = (state >> 5) & 7
target = NM_ENC[i] ^ (state & 0xFF)
input[i] = ROR8(target, shift) ^ (state & 0xFF)
state  = (state ^ input[i]) * 0x01000193


Locating NM_ENC needs no symbol. Slide a 36-byte window across the whole binary and invert the schedule against every offset, keeping the one candidate whose recovered bytes are entirely printable ASCII. That candidate is unique, and it lands exactly on the real .rodata table. It recovers the 36-byte password n4nom1tes_eat_y0ur_symb0lic_execut0r.

Feeding that password back to the binary satisfies every trap in check_coil(), so main() reaches unseal_and_print(). That function derives a keystream from the password and the final key state left over after all 36 bytes were checked, then XORs it against an encrypted flag blob (FLAG_ENC) also sitting in .rodata. The flag is never stored in plaintext anywhere in the binary. It only exists once the correct password has driven the key schedule to the right final state.

Getting the Flag

With the password recovered, run it against the binary directly:

$ printf 'n4nom1tes_eat_y0ur_symb0lic_execut0r' | ./coilvm
SkillBit{n4nom1tes_eat_y0ur_symb0lic_execut0r}

‍Solve Script

The window scan and the inversion above are automated in the following solve script:

#!/usr/bin/env python3
"""
CoilVM Solver
=============
Recovers the 36-byte password gating the CoilVM nanomite crackme, then runs
the target binary with it to unseal the embedded flag.

CoilVM's check_coil() is nothing but int3 (0xCC) software breakpoints; the
real comparison logic lives in the SIGTRAP handler, which decodes each
target byte with a running FNV-1a key that is re-keyed by every accepted
input byte. That makes the "constraint system" a moving target (block i's
target only decodes correctly once bytes 0..i-1 are already known), so
instead of building a flat system for a solver, we emulate the key schedule
forward and solve each byte in lockstep as we go.

This is an offline, static crackme: there is no live service to connect to,
just the downloaded binary.

Usage:
  python3 solve.py ./coilvm
  python3 solve.py ./coilvm --no-run   # only recover the password, skip running the binary
"""

import argparse
import re
import subprocess

from rich.console import Console

console = Console()

S0 = 0x811C9DC5  # FNV-1a 32-bit offset basis, the handler's initial `state`
FNV_PRIME = 0x01000193
MASK32 = 0xFFFFFFFF
PW_LEN = 36


def ror8(x: int, k: int) -> int:
    k &= 7
    return ((x >> k) | (x << (8 - k))) & 0xFF if k else x & 0xFF


def invert_schedule(nm_enc: bytes) -> bytes:
    """Walk the handler's key schedule forward, solving each byte in lockstep.

    The handler checks ROL8(in ^ (state&0xFF), (state>>5)&7) == NM_ENC[i] ^ (state&0xFF).
    That's invertible: given NM_ENC[i] and the current state we can solve for `in`
    directly, then fold it into `state` exactly like the handler does before moving
    on to byte i+1.
    """
    state = S0
    out = bytearray()
    for enc_byte in nm_enc:
        shift = (state >> 5) & 7
        target = (enc_byte ^ (state & 0xFF)) & 0xFF
        b = ror8(target, shift) ^ (state & 0xFF)
        out.append(b)
        state = ((state ^ b) * FNV_PRIME) & MASK32
    return bytes(out)


def recover_password(binary: bytes) -> bytes:
    """Locate the 36-byte NM_ENC table by sliding a window across the binary.

    The binary is stripped, so there's no symbol to locate NM_ENC by. Instead,
    the correct window is the only one whose inverted schedule comes out entirely
    printable ASCII. The nanomite table is build-time data derived from an
    ASCII password, so that property uniquely identifies it.
    """
    for off in range(len(binary) - PW_LEN):
        candidate = invert_schedule(binary[off : off + PW_LEN])
        if all(0x20 <= c <= 0x7E for c in candidate):
            console.print(
                f"[bold cyan][*][/bold cyan] candidate NM_ENC @ 0x{off:x} -> {candidate!r}"
            )
            return candidate
    raise RuntimeError(
        "no printable password recovered: schedule constants may differ"
    )


def get_flag(path: str, password: bytes) -> str:
    """Feed the recovered password to the target and read the unsealed flag back."""
    proc = subprocess.run([path], input=password, capture_output=True, timeout=30)
    output = proc.stdout.decode(errors="replace").strip()
    match = re.search(r"SkillBit\{.*?\}", output)
    if not match:
        raise RuntimeError(f"binary did not unseal a flag, got: {output!r}")
    return match.group(0)


def main(path: str, run_binary: bool) -> None:
    console.print(f"[bold cyan][*][/bold cyan] Reversing [underline]{path}[/underline]")
    binary = open(path, "rb").read()

    password = recover_password(binary)
    console.print(
        f"[bold green][+][/bold green] Recovered password: {password.decode()}"
    )

    if not run_binary:
        return

    flag = get_flag(path, password)
    console.print(f"[bold green][+] Flag:[/bold green] {flag}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="coilvm solver")
    parser.add_argument("binary", help="Path to the coilvm binary")
    parser.add_argument(
        "--no-run",
        action="store_true",
        help="Only recover the password, skip executing the binary",
    )
    args = parser.parse_args()

    main(args.binary, not args.no_run)