Basic Reverse Engineering Vol. 5: x86 32-bit Stack Allocation V.S. x64

You will see the comparison table between x86 and x64, ground truth from Visual Studio 2019 debugger. I also add my thought about how this relates to reverse engineer Rust assembly.

Table of Contents

Part 1. x86 32-bit stack allocation

• 1.0 Settings for Visual Studio 2019
• 1.1 The source code
• 1.2 What MSVC 2019 actually produces: x86–32 debug
• 1.3 The stack layout inside func at [ebp+8] through [ebp+18h]
• 1.4 The push ecx idiom — MSVC only
• 1.5 The x64 MS ABI equivalent — same source, Windows x64 MSVC
• 1.6 Head-to-head comparison
• 1.7 The EBP chain: what the maximum stack diagram is really showing

Part 2. Further thought

• 2.1 Core Concept
• 2.2 cdecl vs stdcall: The One Key Difference
• 2.3 Why They Matter for RE and Malware Research

References

Part 1. x86 32-bit stack allocation

This is a learning note based on the Free (awesome) online course: OST2 - Architecture 1001: x86–64 Aseembly. You can also see my research about reverse engineering Rust and further info driven from the content.

1.0 Settings for Visual Studio 2019

The specific settings that produce this behaviour in MSVC 2019 are almost certainly:

Linker settings:

• /DYNAMICBASE:NO — disables ASLR, fixes the image base
• /FIXED — tells the linker the image cannot be rebased
• /BASE:0x140000000 — explicitly sets the preferred load address (default for x64 PE)

Compiler settings that explain the frame pointer omission:

• /Oy or the equivalent optimisation level that omits frame pointers even at /Od — or it could simply be that MSVC 2019 omits RBP for leaf functions by default in x64 regardless of debug settings, which is standard MS ABI behaviour

A leaf function is a function that does not call any other functions.

Build settings that keep addresses stable across runs:

• Debug configuration with incremental linking disabled (/INCREMENTAL:NO)
• Possibly /ZI replaced with /Zi to avoid edit-and-continue padding

This also explains why the 32-bit example lands at 00401000 and 00401010 so cleanly — that is the default x86 image base (0x400000) with no padding, which only happens when you strip out the extra debug alignment that incremental linking normally inserts.

1.1 The source code

Both examples in the slides use the same C source. The first is a simple zero-parameter function; the second adds five integer parameters and a local variable. Both compiled as x86–32 debug with MSVC 2019 (/Od, no optimisation).

Example 1 — simple return

int func() {
    return 0xbeef;
}
int main() {
    func();
    return 0xf00d;
}

example 2 — five parameters + local variable

int func(int a, int b, int c, int d, int e) {
    int i = a + b - c + d - e;
    return i;
}
int main() {
    return func(0x11, 0x22, 0x33, 0x44, 0x55);
}

1.2 What MSVC 2019 actually produces: x86–32 debug

These are transcribed exactly from the Visual Studio debugger screenshots. Every address, mnemonic, and operand is taken directly from the screen — not generated or inferred.

Example 1: simple func + main (32-bit)

func
00401000  push   ebp           ; save caller's EBP
00401001  mov    ebp, esp      ; anchor new frame — EBP = ESP
00401003  mov    eax, 0BEEFh   ; return value into EAX
00401008  pop    ebp           ; restore caller's EBP
00401009  ret                  ; plain ret — cdecl, caller cleans

main
00401010  push   ebp
00401011  mov    ebp, esp
00401013  call   func (0401000h)
00401018  mov    eax, 0F00Dh
0040101D  pop    ebp
0040101E  ret

Note: No add esp, N after the call — because func takes zero parameters, there is nothing on the stack to clean up. The frame pointer pair (push ebp / mov ebp, esp) appears even in a zero-parameter function in MSVC debug mode.

Example 2: five-parameter func + main (32-bit) : from debugger

func — prologue + local allocation + arithmetic
00401000  push   ebp                      ; save caller EBP
00401001  mov    ebp, esp               ; anchor frame
00401003  push   ecx                      ; MSVC idiom: allocate 4 bytes for int i
           ; NOT a callee-save — ECX value discarded
           ; equivalent to sub esp, 4 but 1 byte shorter
00401004  mov    eax, dword ptr [a]       ; load a = [ebp+8]
00401007  add    eax, dword ptr [b]       ; + b = [ebp+0Ch]
0040100A  sub    eax, dword ptr [c]       ; - c = [ebp+10h]
0040100D  add    eax, dword ptr [d]       ; + d = [ebp+14h]
00401010  sub    eax, dword ptr [e]       ; - e = [ebp+18h]
00401013  mov    dword ptr [i], eax       ; store result → [ebp-4]
           ; return i;
00401016  mov    eax, dword ptr [i]       ; return value in EAX
00401019  mov    esp, ebp               ; epilogue step 1
0040101B  pop    ebp                      ; epilogue step 2: restore caller EBP
0040101C  ret                           ; plain ret — cdecl

main - push args right-to-left, cdecl cleanup
00401020  push   ebp
00401021  mov    ebp, esp
00401023  push   55h                      ; arg5=e, pushed first
00401025  push   44h                      ; arg4=d
00401027  push   33h                      ; arg3=c
00401029  push   22h                      ; arg2=b
0040102B  push   11h                      ; arg1=a, pushed last → [ebp+8] in func
0040102D  call   func (0401000h)
00401032  add    esp, 14h               ; cdecl cleanup: 5 × 4 = 20 bytes
00401035  pop    ebp
00401036  ret

1.3 The stack layout inside func at [ebp+8] through [ebp+18h]

After the prologue completes (push ebp / mov ebp, esp / push ecx), the stack looks exactly as the OST2 slide diagrams show. EBP is fixed; all accesses are stable offsets from it for the entire function lifetime.

1.4 The push ecx idiom: MSVC only

The most important thing to understand about line 00401003 is what it is not: it is not saving ECX as a callee-saved register. The OST2 slide makes this explicit with the annotation: "Not a callee-save! It's a VS-ism for allocating a single int worth of space for a local variable."

MSVC uses push ecx as a one-byte instruction to decrement ESP by 4, allocating stack space for int i. The value in ECX at that point is discarded — it is never read back. The equivalent but longer form is sub esp, 4 (three bytes). For multiple locals MSVC does use sub esp, N, but for a single 4-byte local the push ecx trick is common in debug builds.

RE trap: If you see push ecx immediately after the frame pointer prologue and assume ECX is being callee-saved, you will incorrectly expect a corresponding pop ecx in the epilogue. There is none. The epilogue uses mov esp, ebp to discard the entire local frame including this slot, then pop ebp. GCC never emits this pattern — it always uses explicit sub esp, N. Seeing push ecx/edx after the prologue is therefore a reliable MSVC compiler fingerprint.

1.5 The x64 MS ABI equivalent: same source, Windows x64 MSVC

For direct comparison, here is what MSVC produces for exactly the same source compiled as x64 on Windows. This is the Microsoft x64 ABI — RCX, RDX, R8, R9 for the first four integer arguments, then the stack, with a mandatory 32-byte shadow space.

func():

; prologue + parameter spill
140001000  mov    dword ptr [a], r9d       ; spill arg4=d from R9  → [rsp+18h]
140001005  mov    dword ptr [rsp+18h], r8d ; spill arg3=c from R8  → shadow+8
14000100A  mov    dword ptr [rsp+10h], edx ; spill arg2=b from RDX → shadow+8
14000100E  mov    dword ptr [rsp+8], ecx   ; spill arg1=a from RCX → shadow base
140001012  sub    rsp, 18h                  ; allocate locals (24 bytes)

; int i = a + b - c + d - e
140001016  mov    eax, dword ptr [b]        ; eax = b
14000101A  mov    ecx, dword ptr [a]        ; ecx = a
14000101E  add    ecx, eax                  ; ecx = a+b
140001020  mov    eax, ecx                  ; eax = a+b
140001022  sub    eax, dword ptr [c]        ; eax = a+b-c
140001026  add    eax, dword ptr [d]        ; eax = a+b-c+d
14000102A  sub    eax, dword ptr [e]        ; eax = a+b-c+d-e = i
14000102E  mov    dword ptr [rsp], eax      ; store i at [rsp+0]

; return i
140001031  mov    eax, dword ptr [rsp]      ; return value in EAX
140001034  add    rsp, 18h                  ; restore stack
140001038  ret

main():

; main prologue + call
140001040  sub    rsp, 38h                      ; 56 bytes: shadow(32)+arg5(8)+align(16)
           ; return func(0x11, 0x22, 0x33, 0x44, 0x55)
140001044  mov    dword ptr [rsp+20h], 55h      ; arg5=e → stack above shadow
14000104C  mov    r9d, 44h                      ; arg4=d → R9
140001052  mov    r8d, 33h                      ; arg3=c → R8
140001058  mov    edx, 22h                      ; arg2=b → RDX
14000105D  mov    ecx, 11h                      ; arg1=a → RCX
140001062  call   func (0140001000h)
140001067  add    rsp, 38h                      ; restore 56 bytes
14000106B  ret

Transparency note: The 32-bit assembly above (example 2) is directly from the MSVC 2019 debugger. The x64 assembly is my reconstruction of what MSVC generates for the same source. It follows the MS x64 ABI specification exactly, but you should verify it yourself by compiling with cl /Od /Zi targeting x64.

x64 stack layout inside func

1.6 Head-to-head comparison

1.7 The EBP chain: what the maximum stack diagram is really showing

The OST2 “maximum stack diagram” slides show three nested frames: main → foo → bar. The key insight is that every "saved EBP" cell is a 4-byte pointer to the previous frame's EBP. This forms a linked list through the stack:

; at the moment bar() is executing:
bar's  [ebp+0]  →  foo's EBP   (0019fee4 in the slide)
foo's  [ebp+0]  →  main's EBP  (0019ff04 in the slide)
main's [ebp+0]  →  invoke_main's EBP

This is the mechanism Visual Studio’s call stack window uses to display the full call chain. It is also why frame pointer omission (/Oy in MSVC, -fomit-frame-pointer in GCC) breaks debugger stack unwinding — without saved EBP values, the chain cannot be followed.

The slide also notes that function parameters are accessed as [ebp+offset] and local variables as [ebp-offset], with the GCC exception that locals may be referenced as [esp+offset] when the frame pointer is omitted. MSVC debug builds always use EBP-relative addressing.

Part 2. 32-bit Calling Conventions:

2.1 The Core Concept

In 32-bit x86, there is no register-based parameter passing (unlike x64’s RCX/RDX/R8/R9 or System V’s RDI/RSI/RDX…). All parameters go on the stack via PUSH instructions, right-to-left, so that when you dereference [ESP+4] inside the function, you hit the first (leftmost) argument.

; Calling f(1, 2, 3) in cdecl
push 3          ; last arg pushed first
push 2
push 1
call f
add esp, 12     ; caller cleans up (cdecl only)

Inside f, the stack looks like:

[ESP+0]  → return address
[ESP+4]  → arg1  (leftmost, lowest address)
[ESP+8]  → arg2
[ESP+12] → arg3

This is why the slide says “leftmost parameter ends up at the lowest address” — it was pushed last, so it sits closest to the top of stack.

2.2 cdecl vs stdcall: The One Key Difference

The ret N in stdcall is the giveaway in disassembly — e.g. ret 0x8 means the callee pops 8 bytes of args itself.

2.3 Why This Matters for Rust

1. Legacy malware and shellcode is almost entirely 32-bit

The BODMAS dataset comprises 57,293 malware samples across 581 families [Yang et al., DLS’21]; however, the dataset paper does not report PE architecture breakdown. Whether 32-bit samples constitute a significant proportion remains uncharacterised in the published metadata, and would require direct binary analysis to establish. Misidentifying the calling convention breaks your stack frame reconstruction entirely. If you apply cdecl assumptions to a stdcall function, your arg offsets are wrong from the first instruction.

2. Rust cross-compiles to 32-bit targets

i686-pc-windows-msvc is a legitimate target. Documented Rust-based malware — including RALord, SPICA, and Myth Stealer — exploits the general complexity of Rust binaries to hinder analysis, rather than architecture-specific targeting.

Whilst 32-bit Rust targets such as i686-pc-windows-msvc receive comparatively less tooling attention than their 64-bit counterparts, no published threat intelligence report documents adversaries deliberately selecting 32-bit Rust targets for evasion purposes. Whether this represents an uninvestigated attack surface remains an open research question. Recognising ret N vs add esp, N tells you immediately which convention the compiler chose.

3. Stack-based arg passing is simpler to recover than register-based

No paper directly benchmarks argument recovery accuracy 32-bit vs 64-bit head-to-head. What the literature does support is a narrower, more honest version of the claim:

Binary-level type recovery is inherently speculative arXiv regardless of architecture. The relevant architectural observation which is supportable, comes from the ABI difference itself:

x64 uses registers for the first four arguments Microsoft Learn, whereas in 32-bit cdecl/stdcall, all arguments are stack-passed. This is an architectural fact, not a measured accuracy claim.

The stack-only parameter passing of 32-bit cdecl/stdcall leaves a more structurally consistent argument trace than the register-based x64 ABI, where argument liveness is subject to compiler optimisation, inlining, and register reuse. However, no published benchmark has directly measured argument recovery accuracy across these two ABIs.

4. Windows API detection in malware

If you see call [EAX] preceded by a series of pushes and no add esp after the call, you're looking at a stdcall indirect call, a classic pattern for dynamic API resolution (GetProcAddress → stdcall target). This is bread-and-butter for malware that walks the PEB to resolve imports at runtime.

5. Mixed binaries

MSVC sometimes emits both conventions in the same binary (e.g., internal functions use cdecl, exported Win32-compatible functions use stdcall). Binary Ninja’s type system handles this via __cdecl / __stdcall annotations, worth verifying these are applied correctly when you're doing automated API enumeration in Binja plugin work (or any other disassembler).

References

• Andriesse et al., An In-Depth Analysis of Disassembly on Full-Scale x86/x64 Binaries, USENIX Security 2016. https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_andriesse.pdf — covers disassembly accuracy degradation at higher optimisation levels, relevant context
• Pang et al., SoK: All You Ever Wanted to Know About x86/x64 Binary Disassembly, IEEE S&P 2021. https://arxiv.org/abs/2007.14266— covers function recovery accuracy across x86/x64
• Vaidya et al., Assessing the Effectiveness of Binary-Level CFI Techniques, arXiv 2024 . https://arxiv.org/pdf/2401.07148 — directly states binary-level type recovery is inherently speculative
• Fog, Calling Conventions for Different C++ Compilers and Operating Systems. https://www.agner.org/optimize/calling_conventions.pdf— the authoritative reference for ABI details across 32/64-bit