Interesting instructions in Rust binary: lock cmpxchg

The story starts when I reverse engineer a “normal” Rust binary (release build, MCVC x64 PE), also inspected a tab “Tags” in Binary Ninja.

It says…

Tabs — Unresolved Indirect Control Flow (13)

As we may (or not) know, function boundary identification is notoriously hard in Rust binaries (I think it’s still is in C++ binaries too). Not surprised I saw them. For the Reverse Engineering workflow, the unresolved indirect jumps can be the most interesting targets. Each one is a potential trait object dispatch site. To resolve them:

1. Navigate to the jmp rax site
2. Look backwards for where rax was loaded: typically mov rax, [some_ptr + offset] where some_ptr points into a vtable
3. Find the vtable in .data.rel.ro and read the function pointer array
4. Cross-reference with panic strings or symbol hashes to identify the trait

But what attracts me and grabbed my attention is the unusal instruction lock cmpxchg (see the figure below)…

Ummm… interesting.

You can have a look at some disucssion and resources, like:

• https://stackoverflow.com/questions/59020823/how-does-lock-cmpxchg-work-in-assembly
• https://medium.com/@m-ibrahim.research/atomic-moves-lock-free-programming-straight-from-the-cpu-446f0ffe83c6
• https://heather.cs.ucdavis.edu/matloff/public_html/50/PLN/lock.pdf

As described in this course material:

Operation of the ‘cmpxchg’ instruction is described (on 3 pages) in Volume 2A of Intel Manual.

First thing first, this is nothing to do with the source code. It comes entirely from the stdlib’s futex::Once::call, triggered indirectly by println! initialising the stdout singleton on its first call.

use std::hint::black_box;

trait Speak {
    fn say_hello(&self);
}

struct Human;
struct Robot;

impl Speak for Human {
    fn say_hello(&self) {
        println!("Hi!");
    }
}

impl Speak for Robot {
    fn say_hello(&self) {
        println!("Beep boop!");
    }
}

#[inline(never)]
fn greet(speaker: &dyn Speak) {
    speaker.say_hello();
}

fn main() {
    let h = Human;
    let r = Robot;

    greet(&h);
    greet(&r);

    black_box(&h);
    black_box(&r);
}

Lock-Free Programming

Let’s have a quick recap before discussing this instruction…

The problem it solves

In concurrent programming, multiple threads share data. The naive solution is a mutex:

let data = Mutex::new(0u64);
// Thread A:
let mut guard = data.lock().unwrap();  // blocks if Thread B holds it
*guard += 1;
// lock released on drop

Mutexes work, but they have costs:

• Blocking: a thread that cannot acquire the lock goes to sleep, requiring OS scheduler involvement (syscall, context switch)
• Priority inversion: a low-priority thread holding the lock blocks a high-priority thread
• Deadlock: two threads each waiting for a lock the other holds
• Convoying: threads queue behind the lock holder even if the critical section is trivially short

Lock-free programming eliminates the lock entirely. Instead of serialising access through mutual exclusion, it uses hardware atomic operations that the CPU guarantees are indivisible.

The hardware foundation: atomicity

On x86–64, certain operations are guaranteed atomic by the ISA:

• Aligned reads and writes up to 64 bits are naturally atomic (no tearing)
• The lock prefix makes a read-modify-write sequence atomic: lock add, lock xchg, lock cmpxchg

“Atomic” means: no other core can observe the memory location in an intermediate state. The operation either has happened completely or not at all, from every core’s perspective.

The key primitive: Compare-and-Swap (CAS)

Everything in lock-free programming is built on CAS. In Rust:

rust

use std::sync::atomic::{AtomicU64, Ordering};
let counter = AtomicU64::new(0);
// CAS: "if counter == expected, set it to desired, return Ok(desired)"
//      "if counter != expected, return Err(actual_value)"
counter.compare_exchange(expected, desired, success_ord, fail_ord);

The canonical lock-free update pattern:

loop {
    let current = counter.load(Ordering::Acquire);
    let next = current + 1;
    match counter.compare_exchange(current, next, Ordering::Release, Ordering::Relaxed) {
        Ok(_) => break,       // CAS succeeded, we won the race
        Err(_) => continue,   // another thread changed it first, retry
    }
}

This is exactly the loop you saw earlier in sub_140003b10. No mutex, no OS call, no sleeping.

What lock cmpxchg is

cmpxchg = Compare and Exchange. It is the x86 atomic compare-and-swap primitive.

The semantics in one line:

If [mem] equals rax, write rdx into [mem] and set ZF=1. Otherwise load [mem] into rax and set ZF=0. Atomically.

The lock prefix makes the entire operation a single atomic bus transaction: it asserts the cache-line lock on the target memory address for the duration, preventing any other core from reading or writing that address between the compare and the exchange.

Without lock, on a multicore system, another thread could modify [mem] between the compare and the exchange, creating a race. With lock, the entire read-modify-write is indivisible.

The full instruction here

lock cmpxchg qword [rel data_140020208], rdx

Unpacked:

Component Meaning lock Atomically hold the cache line cmpxchg Compare-and-swap qword 64-bit operation [rel data_140020208] RIP-relative address: a global variable in .data/.bss rdx The new value to write if the compare succeeds (implicit) rax The expected current value (loaded earlier)

So the operation is: *if `data_140020208 == rax, then storerdx` there atomically.**

What the full loop is doing

Looking at both images together, the HLIL makes it clear. Let me annotate the disassembly against the HLIL (High Level IL in Binary Ninja):

; ── Load current value of the global ──────────────────────────
140003b50   mov rax, qword [rel data_140020208]   ; rax_2 = *global

; ── Spin loop top ─────────────────────────────────────────────
140003b60   cmp rax, 0xffffffffffffffff           ; if rax_2 == -1 (sentinel)
140003b64   je  0x140003bd3                       ;   → call sub_140016790 (noreturn: panic/abort)
140003b66   lea rdx, [rax+0x1]                    ; rdx = rax_2 + 1  (desired new value)
140003b6a   lock cmpxchg qword [rel data_140020208], rdx
            ; attempt: if *global == rax, write rdx, ZF=1
            ; else:    rax = *global (reload), ZF=0
140003b73   jne 0x140003b60                       ; if ZF=0 (CAS failed), retry
; ── CAS succeeded ─────────────────────────────────────────────
140003b75   mov qword [rcx], rdx                  ; *(rax_1 + 0x38) = rdx

This is a compare-and-swap spin loop: a standard lock-free increment.

In pseudocode:

HIL in Binary Ninja

Pseudo C in Binary Ninja

loop {
    let current = global.load(Relaxed);   // rax
    if current == -1 { panic!(); }        // sentinel: poisoned/uninitialized
    let next = current + 1;               // rdx
    if global.compare_exchange(current, next).is_ok() {
        break;
    }
    // else: another thread changed global, reload rax and retry
}

The jne after lock cmpxchg is checking the zero flag: if ZF=0, the CAS failed (someone else modified the global between our load and our CAS attempt), so we jump back to reload and retry.

What this pattern represents in Rust

The context gives it away. Look at what happens after the loop succeeds:

140003b78   mov qword [rel data_140020200], rdx   ; store thread count / handle
140003b88   call AddVectoredExceptionHandler       ; VEH registration
140003b8e   mov dword [rbp-0x38], 0x5000          ; StackSizeInBytes = 0x5000
140003b99   call SetThreadStackGuarantee
140003b9f   call GetCurrentThread
140003bac   lea rdx, [rel data_14001907a]         ; u"main"

The TLS access at the top (gs:0x58 → ThreadLocalStoragePointer), the CAS on a global counter, followed by AddVectoredExceptionHandler and stack guarantee setup, and the string u"main" — this is the Rust runtime initialisation sequence. Specifically this is std::rt::lang_start or its inner worker, which runs once per thread and uses a CAS-protected counter to track initialisation state.

The specific pattern (global atomically incremented, sentinel value -1 triggers a noreturn) is Rust's once-cell / lazy initialisation pattern, likely std::sync::Once or the internal lang_start_internal synchronisation that ensures the runtime (panic handler, VEH, stack guarantee) is initialised exactly once even under concurrent entry.

The HLIL renders it cleanly:

rax_2 = data_140020208   // load global
do {
    if (rax_2 == -1) { sub_140016790(); noreturn }  // sentinel check
    rdx = rax_2 + 1
    if (rax_2 == data_140020208) {                  // CAS success path
        data_140020208 = rdx
        z_1 = true
    } else {
        rax_2 = data_140020208                      // CAS failure: reload
        z_1 = false
    }
} while (not z_1)

Binary Ninja’s HLIL has correctly identified the lock cmpxchg + jne as a do-while CAS loop and modelled the success/failure branches.

Why this is a release build indicator (supporting evidence)

A debug build would not have this pattern here in the same way. In a debug build:

• The std::sync::Once or equivalent would appear as an explicit call to the unoptimised Once::call_once function, not as an inlined CAS loop
• The TLS access would go through several more indirection layers with intervening stack spills
• The whole initialisation sequence would be spread across multiple non-inlined call frames

The fact that the entire CAS loop, VEH registration, stack guarantee, and thread naming is all inlined into a single flat function body is a strong optimisation signal consistent with a release build at opt-level 2 or 3.

So, where does it come from…?

Where lock cmpxchg comes from in your binary

Step 1: mod.rs routes x86_64-pc-windows-msvc to futex.rs

The dispatch table in library/std/src/sys/sync/once/mod.rs uses cfg_select!:

cfg_select! {
    any(
        all(target_os = "windows", not(target_vendor="win7")),  // modern Windows
        target_os = "linux",
        ...
    ) => {
        mod futex;   // x86_64-pc-windows-msvc lands here
    }
    any(
        windows,     // older/fallback Windows
        ...
    ) => {
        mod queue;
    }
}

x86_64-pc-windows-msvc has target_os = "windows" and target_vendor = "pc". Since "pc" != "win7", the condition not(target_vendor="win7") is true, so the first arm matches and the binary links against futex.rs.

Reference: [https://doc.rust-lang.org/src/std/sys/sync/once/mod.rs.html](https://doc.rust-lang.org/src/std/sys/sync/once/mod.rs.html)

Step 2: futex.rs contains compare_exchange_weak, which compiles to lock cmpxchg

Inside futex::Once::call(), the slow path for first initialisation:

INCOMPLETE | POISONED => {
    // Try to register the current thread as the one running.
    let next = RUNNING + if queued { QUEUED } else { 0 };
    if let Err(new) = self.state_and_queued.compare_exchange_weak(
        state_and_queued,   // expected: current state (INCOMPLETE = 3)
        next,               // desired: RUNNING (= 1)
        Acquire,
        Acquire,
    ) {
        state_and_queued = new;
        continue;           // CAS failed, retry
    }
    // ... run the initialiser closure
}

On x86–64, LLVM lowers compare_exchange_weak on an atomic integer with Acquire ordering to lock cmpxchg. The surrounding retry loop (continue) is the spin you see in the disassembly.

Reference: [https://doc.rust-lang.org/src/std/sys/sync/once/futex.rs.html](https://doc.rust-lang.org/src/std/sys/sync/once/futex.rs.html)

Step 3: the call chain from your source code

The source code has println!("Hi!") inside Human::say_hello. That expands to a call to std::io::_print, which calls std::io::stdout(). Inside stdout():

// library/std/src/io/stdio.rs
static STDOUT: OnceLock<ReentrantLock<RefCell<LineWriter<StdoutRaw>>>> = OnceLock::new();

pub fn stdout() -> Stdout {
    Stdout { inner: STDOUT.get_or_init(|| { ... }) }
}

OnceLock::get_or_init delegates to Once::call_once, which on x86_64-pc-windows-msvc routes to futex::Once::call. On the very first call to println! in the process, the state is INCOMPLETE (= 3), so the INCOMPLETE | POISONED arm fires and executes compare_exchange_weak, producing the lock cmpxchg at 140003b6a.

Reference: [https://doc.rust-lang.org/src/std/io/stdio.rs.html](https://doc.rust-lang.org/src/std/io/stdio.rs.html)

What the assembly maps to?

; rax = load of Once state (INCOMPLETE = 0)
140003b50   mov rax, [rel data_140020208]
; cmp rax, -1 = check for POISONED sentinel
140003b60   cmp rax, 0xffffffffffffffff
140003b64   je  0x140003bd3            ; → panic (poisoned)
; rdx = rax + 1 = RUNNING (0 + 1 = 1)
140003b66   lea rdx, [rax+0x1]
; compare_exchange(INCOMPLETE, RUNNING, SeqCst)
140003b6a   lock cmpxchg [rel data_140020208], rdx
; if ZF=0 (CAS failed, state changed), retry
140003b73   jne 0x140003b60

This can be:

// queue.rs
let old = self.state.compare_and_swap(state, RUNNING, Ordering::SeqCst);
if old != state { state = old; continue }

The -1 / 0xffffffffffffffff check is the POISONED sentinel (or a waiter-pointer sentinel in the queue.rs encoding; the upper bits of the AtomicUsize encode the waiter linked list head in the RUNNING state, so values larger than COMPLETE that are not a valid pointer trigger the abort path).

Summary of lock cmpxchg mechanics

Before:          rax = expected_value
                 rdx = new_value
                 mem = some_current_value

Case A (success): mem == rax
                 → mem becomes rdx
                 → ZF = 1
                 → jne not taken: loop exits
Case B (failure): mem != rax
                 → rax loaded with actual mem value
                 → ZF = 0
                 → jne taken: loop retries with fresh rax

The loop continues retrying until the CAS succeeds, i.e. until no other thread has modified the global between this thread’s load and its attempted store. Under low contention this succeeds on the first or second attempt. Under high contention it spins. This is the foundation of all lock-free data structures on x86.