Hidden Gems

For Cyhub CTF 2025, I created a multi-stage reverse engineering challenge that combines modern and classic RE techniques. Hidden Gems is built with Rust and involves decrypting a hidden ELF binary embedded within the main program. The challenge required players to understand binary analysis, encryption schemes, and multi-stage exploitation.

Challenge Description

Players are provided with the hidd3n_g3ms binary. The goal is to find the flag hidden through two layers of obfuscation.

Category: Reverse Engineering
Difficulty: Medium
Flag Format: cyhub{xxxxx}

Challenge Overview

This challenge involves two main stages:

1. Stage 1 - Vault Unsealing: Analyze the Rust binary to find the correct "vault key" that will decrypt a hidden ELF binary
2. Stage 2 - 2FA Bypass: Analyze the decrypted binary to extract the 2FA key and construct the final flag

The twist? The binary is written in Rust, which produces different assembly patterns compared to traditional C/C++ binaries. Additionally, the second-stage binary is XOR-encrypted and embedded within the first binary, only being decrypted and executed in-memory when the correct vault key is provided.

Solution Walkthrough

Stage 1: Finding the Vault Key

Step 1: Static Analysis

Use a disassembler like Ghidra, IDA Pro, or radare2 to analyze the hidd3n_g3ms binary. Since it's a Rust binary, you'll notice some differences in how functions are structured and named compared to typical C binaries.

Step 2: Locate the unseal function

This function contains the vault key validation logic. In the disassembled code (src/main.rs:97), you'll find:

auVar2 = __udivti3(uStack_b8,uStack_b0,0x13d9415c859454c,0);
if (auVar2._0_8_ == 0x539 && auVar2._8_8_ == 0) {

This is a 128-bit integer division operation that's equivalent to:

if n / 89390388893795660 == 1337 {
    // vault unseals
}

Step 3: Calculate the key (Intended Solution)

The intended approach is to use the correct key I designed for the challenge:

n = 119514949951004811051

Convert this number to ASCII:

• 119514949951004811051 represents the bytes: [119, 51, 49, 49, 95, 100, 48, 110, 51]
• Which translates to: w311_d0n3

However, if you try the straightforward mathematical approach by solving n = 89390388893795660 * 1337, you get:

89390388893795660 * 1337 = 119514949951004797420

This doesn't match the intended key! Why? Because of a bug in my challenge design.

Step 4: The Floor Division Bug

After the CTF, Hayk pointed out an interesting bug in my implementation. Since the check uses integer floor division:

if n / 89390388893795660 == 1337 {
    // vault unseals
}

Any value in the following range will pass the check:

119514949951004797420 <= n < 119514949951094187780

This is because integer division truncates (floors) the result. So 1337.something becomes 1337.

My intended key 119514949951004811051 falls within this valid range, which is why it worked. But mathematically, the "correct" answer from simple multiplication (89390388893795660 * 1337 = 119514949951004797420) also works!

How players solved it:

Despite this bug, players still managed to find valid keys through manual brute-forcing:

1. They could extract partial ASCII from the numbers (like w311_)
2. By manually trying different printable ASCII characters for the remaining positions
3. They found numbers within the valid range that decoded to readable strings

It's a good reminder that challenge design requires careful attention to edge cases in mathematical operations.

Vault Key: w311_d0n3

Stage 2: Extracting the 2FA Key

Step 1: Run with the vault key

Execute the binary and provide w311_d0n3 when prompted for the vault key.

Step 2: Binary decryption

The program will decrypt an embedded ELF binary and execute it in memory, prompting: "2FA Required (input char by char)".

Step 3: Dynamic analysis with GDB

gdb ./hidd3n_g3ms
(gdb) run
# Enter vault key: w311_d0n3
# When it prompts for 2FA, press CTRL+C

Step 4: Disassemble the decrypted binary

Once stopped, you can disassemble the running code:

pwndbg> disassemble main
Dump of assembler code for function main:
   0x000055555555551a <+0>:     push   rbp
   0x000055555555551b <+1>:     mov    rbp,rsp
   ...
   0x0000555555555541 <+39>:    call   0x555555555504 <l0l>
   0x000055555555554b <+49>:    call   0x55555555550f <l33t>
   0x0000555555555555 <+59>:    call   0x555555555179 <g3ms>
   ...

pwndbg> disassemble g3ms
Dump of assembler code for function g3ms:
   0x0000555555555179 <+0>:     push   rbp
   0x000055555555517a <+1>:     mov    rbp,rsp
   0x000055555555517d <+4>:     sub    rsp,0x10
   ...
   0x00005555555551a5 <+44>:    cmp    al,0x74      # Checking for 't' (116)
   0x00005555555551a7 <+46>:    je     0x5555555551b3
   ...
   0x00005555555551e5 <+108>:   cmp    al,0x34      # Checking for '4' (52)
   ...

Step 5: Extract the 2FA key

By analyzing the comparisons in the g3ms function (from gem.c), you'll find it expects these characters:

2FA Key: t4k3_y0ur_g3ms

Final Flag

The program constructs the final flag by combining both parts:

cyhub{<vault_key>_<2fa_key>}
cyhub{w311_d0n3_t4k3_y0ur_g3ms}

Key Technical Details

• Language: Rust - produces different assembly patterns than C/C++
• Encryption: XOR-based encryption for the embedded ELF
• In-memory execution: The decrypted binary never touches disk
• Division check: 128-bit integer division using __udivti3
• ASCII encoding: The vault key is encoded as a large integer
• Character-by-character validation: The 2FA uses sequential character checks

Tools Recommended

• Ghidra - Free reverse engineering tool by NSA
• IDA Pro - Industry-standard disassembler
• GDB with pwndbg - For dynamic analysis
• radare2 - Open-source RE framework

The challenge source code can be found here