Automated Shellcode Triage and API Hash Resolution

A practical workflow with capa, Binary Ninja, and the OALabs HashDB

A Technical Report on Reducing Time-to-Understanding for Position-Independent Windows Shellcode.

Executive Summary

Position independent shellcode was designed to be deliberately confusing. It lacks an import table, resolves its Windows APIs dynamically using hashes, and obfuscates both its strings and payload with one or more levels of encoding. A reverse engineer working with a blob of such code will see nothing but a block of code devoid of any entry point or symbols against which to orient.

In this article, we outline a repeatable triage process that takes an incomprehensible 60 KB shellcode blob and converts it into a somewhat understood and annotated program during a short session of reverse engineering. This technique makes use of three tools: capa for automatic capability detection, Binary Ninja for static analysis, and OALabs HashDB for resolving the hashes back to actual API function names. The sample we work on is a 32-bit Windows stager that resolves APIs using the ROR13 algorithm, allocates memory, and stagers an embedded executable.

1. Introduction

Malware authors favor shellcode for the early stages of an intrusion because it is small, relocatable, and carries none of the metadata that lets a scanner classify a normal PE file. A stager has three jobs: find the Windows APIs it needs without a loader, decode or decrypt a larger payload, and jump to that payload. Each one is written to resist static reading.

Doing this by hand works, but it is slow. The analyst first finds the code worth reading, then works out the obfuscation, then recovers what every hashed API call points to. The workflow here moves most of that work up front. capa reports what the code can do and where those behaviors sit, so the disassembler opens with a short list of addresses worth visiting. HashDB handles the tedious part: turning 32-bit hash constants back into API names.

The rest of the paper walks the sample through that process, from raw blob to recovered APIs.

2. Toolchain

| Tool | Role in the workflow |
| --- | --- |
| capa (Mandiant) | Automated static capability detection. Maps code features to ATT&CK techniques, MBC behaviors, and named capabilities. |
| **capa rules**  | The open rule set that drives detection. Available at `github.com/mandiant/capa-rules`. |
| **Binary Ninja** | Interactive disassembler and decompiler. Used to read the High Level IL, recover structures, and apply annotations. |
| **capa import-to-bn plugin** | Renders capa's JSON results as comments/markers inside Binary Ninja. |
| **OALabs HashDB plugin** | Identifies the hashing algorithm and resolves API hashes against the community HashDB lookup service. |

All analysis below was performed statically. The sample was never executed.

3. Sample Overview

The subject is a raw shellcode blob extracted from an earlier unpacking stage. It has no PE header, so file-type identification tools report it as unknown data. It is 32-bit (i386) code intended to run on Windows.

| Property | Value |
| --- | --- |
| **MD5** | `a7c3f0e6e7b6f1a0147b27a7e52088d2` |
| **SHA-1** | `df7205f168cb0ae912e38875823512332631bedf` |
| **SHA-256** | `822872c4e6799dc4c80f8863911387b31fb6f5f32c82b9b75b2b5a2886a147e4` |
| **SSDEEP** | `1536:UwdKlUb+Dm4s9hN1YkPDckM8HsquOBcrqqRTVrdnsqiMSXke:sI4sZ1YkPH1BcGqFVrBsr` |
| **TLSH** | `T16F538D13C707D47AE683407E3517BAB641393D391271E4AEFE878989A9207E176D1F0B` |
| **File type** | unknown (raw shellcode) |
| **Architecture** | i386 / 32-bit |
| **Target platform** | Windows (`windows-x86`) |
| **File size** | 60.00 KB (61,440 bytes) |

Because the blob has no header, the analyst supplies the format (sc32) and, later, the in-memory base address by hand. Establishing those two facts is the precondition for everything that follows.

4. Capability Identification with capa

capa scans a file for low-level features (API references, strings, constants, structural patterns) and matches them against a rule set. Each rule describes a recognizable behavior, so a match is a statement of capability rather than a guess about family. It is well suited to the questions that matter at triage time: does this thing talk to a network, touch the filesystem, or encrypt data, and where in the binary does it do so.

4.1 Baseline run

The shellcode is analyzed with the sc32 format flag and a local copy of the rule set:

capa -f sc32 -r ~/capa-rules/ 550000.shc

capa produces three summary tables: the ATT&CK mapping, the MBC (Malware Behavior Catalog) mapping, and the matched capabilities.

ATT&CK coverage

| Tactic | Technique |
| --- | --- |
| Defense Evasion | Obfuscated Files or Information [T1027] |
| Defense Evasion | Indicator Removal from Tools [T1027.005] |
| Execution | Shared Modules [T1129] |

MBC coverage

| Objective | Behavior |
| --- | --- |
| Anti-Static Analysis | Argument Obfuscation [B0032.020] |
| Anti-Static Analysis | Stack Strings [B0032.017] |
| Cryptography | Encrypt Data::RC4 [C0027.009] |
| Cryptography | Generate Pseudo-random Sequence::RC4 PRGA [C0021.004] |
| Data | Encode Data::Base64 [C0026.001] |
| Data | Encode Data::XOR [C0026.002] |
| Defense Evasion | Encoding-Standard Algorithm [E1027.m02] |
| Execution | Install Additional Program [B0023] |

Matched capabilities

capa’s capability table for the sample. Each row pairs a named behavior with the rule namespace that produced it.

The capability set already tells a coherent story:

| Category | Behavior | What it implies |
| --- | --- | --- |
| Defense Evasion | Stack strings (8 matches) | Strings are built byte-by-byte on the stack at runtime so they never appear in a static scan. |
| Cryptography | RC4, XOR, Base64 | Multiple decode/decrypt routines for unpacking a payload or configuration. |
| Execution | PEB `ldr_data` access (2 matches) | The code resolves APIs by walking loaded modules itself, with no import table. |
| Payload | Embedded PE file | A full EXE or DLL is carried inside the blob, to be dropped or injected. |

In one sentence: a heavily obfuscated Windows shellcode stager built to bypass static analysis, resolve its own APIs, and deploy an embedded executable.

4.2 Locating capabilities with verbose output

The summary says what the code does. The **-v** flag says where, by printing the virtual address of every match:

capa -f sc32 -r ~/capa-rules/ 550000.shc -v

The verbose run also reports the metadata the analyst needs for the disassembler, including the base address 0x690000, a function count of 168, and the Vivisect feature extractor used. The address-bearing matches are the triage map:

contain obfuscated stackstrings (8 matches)   scope: basic block
  0x690FBF  0x695C69  0x696B55  0x697EF1
  0x698941  0x699359  0x69A7A4  0x69AB9B

encode data using Base64                       scope: function
  0x690EA9

encode data using XOR (2 matches)              scope: basic block
  0x691355  0x6913F8

encrypt data using RC4 PRGA                    scope: function
  0x693785

contain an embedded PE file                    scope: file

access PEB ldr_data (2 matches)                scope: basic block
  0x690467  0x690C0C

Each address is a place worth opening in the disassembler. The table below records the intent behind the ones we pursue:

| Address(es) | Capability | What to look for |
| --- | --- | --- |
| `0x690467`, `0x690C0C` | PEB API resolution | The module walk over `PEB->Ldr`. The names or hashes resolved here reveal the API set. |
| `0x690EA9` | Base64 decoder | The buffer passed into this function is the encoded string or config. |
| `0x691355`, `0x6913F8` | XOR loops | Single- or multi-byte XOR keys used to unpack structures or final payloads. |
| `0x693785` | RC4 PRGA | The heavy decryption routine. The end of this function is where decrypted memory can be dumped. |

4.3 Exporting results for automation

The terminal output is fine for a human reading once. For tooling, capa emits structured JSON:

capa -f sc32 -r ~/capa-rules/ 550000.shc -j > capa_sc.json

The JSON export. The same matches, now in a structured form that other tools can consume.

This file becomes the bridge into Binary Ninja in Section 5.5.

5. Static Analysis in Binary Ninja

With a map of interesting addresses in hand, the analysis moves into Binary Ninja.

5.1 Loading the shellcode

A raw blob carries no information about where it expects to live in memory, so the load options are set by hand. The image base address is set to 0x690000 the value capa reported and the platform to windows-x86. Loading at the correct base matters: it makes the virtual addresses from capa line up with the addresses in Binary Ninja, so a capa match at **0x691355** lands at **0x691355** in the disassembler.

Binary Ninja load options. Image base 0x690000, platform windows-x86, predicted architecture x86.

5.2 The XOR decode routine

Taking the first XOR match, **0x691355**, and jumping to it (press g in linear view and paste the address) lands on a tight decode loop, rendered by the decompiler as a **do/while**:

0069137c  do
00691355      int32_t ecx_2 = *edi_2
00691357      edi_2 = &edi_2[1]
0069135a      int32_t ecx_3 = ecx_2 ^ 0x24eedcb9
00691360      *result_1 = ecx_3.b
00691367      result_1 = &result_1[4]
0069136a      uint32_t ecx_4 = ecx_3 u>> 0x10
0069136d      result_1[0xfffffffd] = (ecx_3 u>> 8).b
00691370      result_1[0xfffffffe] = ecx_4.b
00691376      ebx += 1
00691377      result_1[0xffffffff] = (ecx_4 u>> 8).b
0069137c  while (ebx u< esi_7)

The XOR decode loop in Binary Ninja’s High Level IL. The 32-bit key 0x24eedcb9 is applied to each 4-byte word.

The loop uses a fixed 4-byte key, **0x24eedcb9**. Each iteration:

1. Fetch a 4-byte word from the encrypted source pointer (**edi_2**).

2. Decrypt the whole 32-bit word against the key: **ecx_2 ^ 0x24eedcb9**.

3. Unpack the result into four individual bytes written sequentially to the output buffer (**result_1**): the low byte first, then the result shifted right by 8, 16, and 24 bits.

The fixed key is useful beyond this loop. A 32-bit constant tied to a decode routine is a stable hunting pivot it survives recompilation more often than strings do, and it can seed a YARA rule or a retrohunt.

5.3 PEB-based module enumeration

The two **access PEB ldr_data** matches point at the API resolution machinery. Jumping to **0x690467** shows a function whose decompilation Binary Ninja has reconstructed against the Windows structures:

00690467  void* __fastcall sub_690467(int32_t arg1)
0069047b      TEB* fsbase
0069047b      struct _LDR_DATA_TABLE_ENTRY* Flink =
                  fsbase->ProcessEnvironmentBlock->Ldr->InLoadOrderModuleList.Flink
0069050d      while (true)
0069050d          void* DllBase = Flink->DllBase
00690512          if (DllBase == 0) break
00690483          WCHAR* Buffer  = Flink->BaseDllName.Buffer
00690488          int32_t ebx_1  = Flink->BaseDllName.Length.d
0069048b          Flink = Flink->InLoadOrderLinks.Flink
00690494          int32_t ebp_1  = *(*(DllBase + 0x3c) + DllBase + 0x78)

Binary Ninja resolving the PEB structures: the function reads ProcessEnvironmentBlock->Ldr->InLoadOrderModuleList and iterates the loaded modules.

Binary Ninja recovers the PEB structures cleanly, including the dereference chain into **Ldr**. This gives the shellcode access to the in-load-order module list, the doubly linked list of every DLL loaded into the process. Two fields stand out:

• DllBase — the base address of each module, and the starting point for enumerating that module’s exports.
• BaseDllName.Buffer — the address of the module’s name.

Those two fields are exactly what an API-hashing routine needs: a module name to hash and a base address from which to walk the export table. The offset **DllBase + 0x3c** is the classic step to the PE header from the module base, confirming the function is reading export metadata.

5.4 The resolver and its hash arguments

Examining the cross-references to **sub_690467** shows it is called repeatedly, each time with a single hexadecimal argument:

Cross-references to the resolver function. Six call sites, each passing a different 32-bit constant.

00690040  void* eax   = sub_690467(0x0726774c)
0069004e  void* eax_1 = sub_690467(0x7802f749)
0069005c  void* eax_2 = sub_690467(0xe553a458)
00690068  void* eax_3 = sub_690467(0xc38ae110)
00690076  void* eax_4 = sub_690467(0x945cb1af)
00690084  void* eax_5 = sub_690467(0x959e0033)

The resolver called six times in sequence, each with a hard-coded API hash.

The pattern is unmistakable. sub_690467 is an API resolver. Each constant is a precomputed hash of a module-name/API-name pair. The function walks the loaded modules, hashes each export the same way, and returns the address whose hash matches the argument. The return value is stored and later called.

At this point the structure of the program is clear but its meaning is not. Six API addresses are resolved into **eax** through **eax_5**, but the hashes themselves say nothing about which APIs they are. Recovering those names is the subject of Section 6.

5.5 Importing capa annotations into Binary Ninja

Before resolving hashes, the capa JSON from Section 4.3 can be pulled into Binary Ninja so its findings sit alongside the code. The **import-to-bn.py** script is installed as a plugin (see the Binary Ninja plugin documentation), and after a restart the Load capa file option appears in the plugin menu.

The Load capa file option exposed by the import-to-bn plugin after installation.

Loading capa_sc.json annotates the database and logs each rule match at its address:

[Default] Using capa file /Users/kant/Desktop/process_2268/capa_sc.json
[Default] 0x690ea9: encode data using Base64 (data-manipulation/encoding/base64)
[Default] 0x693785: encrypt data using RC4 PRGA (data-manipulation/encryption/rc4)

Binary Ninja’s log showing capa rule matches mapped onto their virtual addresses

These are the same matches from the command-line run, now navigable inside the disassembler. The two tools are looking at the same addresses, and the analyst can pivot between capa’s verdicts and the underlying code without leaving Binary Ninja.

6. Resolving API Hashes with OALabs HashDB

There are several ways to turn a hash constant back into an API name. One can debug the sample and watch which function each call resolves to at runtime. One can search the web for a constant and hope a prior write-up lists it. Both work; both are slow. The OALabs HashDB plugin automates the lookup against a community-maintained database of hashes that malware commonly uses to obfuscate its imports.

The plugin is installed from Binary Ninja’s plugin manager.

The OALabs HashDB plugin in Binary Ninja’s plugin manager. It provides Hunt (algorithm identification) and Hash Lookup (name resolution) actions.

6.1 Identifying the hashing algorithm

Returning to the resolver call sites where the hash constants are passed:

The six hash constants at the resolver call sites, before resolution.

Right-clicking a constant exposes a HashDB submenu with Hash Lookup, Hunt, and algorithm-selection actions.

The HashDB context menu on a selected hash constant.

The first step is Hunt, which asks the database which algorithm could produce the selected value. The hunt returns two candidates at 100% hit rate:

HashDB algorithm hunt. Two matches: Metasploit ROR13 and shl1_add

metasploit   ROR13 hash used in a lot of shellcode.
             https://github.com/rapid7/metasploit-framework/blob/master/external/source/shellcode/windows/x86/src/hash.py
shl1_add     SHIFT LEFT 1 and ADD

The first candidate, Metasploit ROR13, is one of the most widely reused hashing schemes in Windows shellcode. It computes a hash by rotating an accumulator right by 13 bits and adding each character of the API name, combined with a hash of the module name. Its ubiquity makes it the natural first choice, so it is set as the algorithm for this binary.

6.2 Resolving and bulk-importing module hashes

With the algorithm set, Hash Lookup on the first constant (0x0726774c) resolves it to `LoadLibraryA`. Because that function lives in kernel32.dll, the plugin offers to import every function hash from that module at once:

HashDB recognizes LoadLibraryA and offers to bulk-import all kernel32 hashes.

Accepting the bulk import pulls in the full kernel32 hash set as an enum, so subsequent constants from the same module resolve without another network lookup.

6.3 Applying the resolved names

A resolved hash is not yet visible in the code until it is displayed as the matching enum member. Right-clicking the constant and choosing

Display as → Enum Member (shortcut M):

Display as → Enum Member, the step that swaps a raw constant for its symbolic name.

Binary Ninja then offers the enum to apply. LoadLibraryA is already selected on the left; the right pane lists every imported kernel32 hash and its function name. Choosing Select Enum applies it.

Selecting the hashdb_strings enum member for LoadLibraryA.

The constant 0x0726774c is now rendered as **LoadLibraryA** in the decompilation:

The first call site after resolution: sub_690467(LoadLibraryA).

Repeating the lookup for the remaining constants resolves most of them against the **kernel32** set. One constant does not match the **kernel32** import, which means its API lives in a different module. A fresh Hash Lookup on it resolves to NtFlushInstructionCache, a function in **ntdll.dll**:

A constant that fails against kernel32 resolves to NtFlushInstructionCache; HashDB offers to bulk-import the ntdll hashes.

Accepting the **ntdll** import and applying the enum the same way (M, then Select Enum):

Applying the ntdll enum for NtFlushInstructionCache.

resolves the remaining names. The call block now reads as a list of intentions rather than constants:

The resolver call block after applying HashDB enums. The program’s API set is now legible.

7. Results

7.1 Resolved API set

| Hash | Resolved API | Module | Role |
| --- | --- | --- | --- |
| `0x0726774c` | `LoadLibraryA` | kernel32.dll | Load additional libraries by name. |
| `0x7802f749` | `GetProcAddress` | kernel32.dll | Resolve further exports once a library is loaded. |
| `0xe553a458` | `VirtualAlloc` | kernel32.dll | Allocate memory for the decoded payload. |
| `0xc38ae110` | `VirtualProtect`| - | Not present in the imported `kernel32`/`ntdll` sets; candidate for a further lookup. |
| `0x945cb1af` | `NtFlushInstructionCache` | ntdll.dll | Flush the CPU instruction cache before executing freshly written code. |
| `0x959e0033` | `GetNativeSystemInfo` |

All six API hashes were successfully resolved. Combined with the recovered PEB-walking resolver, the resolved API set provides a clear view of the shellcode’s initialization and staging behavior. Replacing raw hash constants with symbolic API names transforms the code from an obfuscated bootstrap routine into a readable execution chain, significantly reducing analyst effort and accelerating behavioral understanding.

7.2 Tracing a resolved API to its use

Resolution is only half the value; the addresses are still called later. Because **VirtualAlloc**’s resolved address is stored in **eax_2**, highlighting that variable and scrolling down reveals where the allocation actually happens:

Highlighting the resolved VirtualAlloc pointer (eax_2) leads to the call site where memory is allocated for the payload.

From here the analysis can continue the same way for each resolved API, following the data flow from resolution to use until the full unpack-and-execute sequence is reconstructed.

8. Behavioral Assessment

The recovered API set, combined with the capabilities capa reported, describes a self-contained payload stager. The four resolved functions form a recognizable sequence:

• LoadLibraryA+GetProcAddress` extend the program’s reach beyond the handful of APIs it bootstraps through the PEB walk. Once these two are in hand, the shellcode can load any DLL and resolve any export it wants, by name.
• VirtualAlloc reserves a region of memory for the decoded payload. For staging shellcode this region is commonly allocated with execute permission so code can be written and then run from it.
• NtFlushInstructionCache is the tell. A program flushes the instruction cache after writing instructions to memory it is about to execute, so the CPU does not run stale cached bytes. Seeing it here is strong evidence that the stager writes code into the buffer it allocated and then jumps to it.

Put together with the embedded PE file and the RC4/XOR/Base64 decode routines, the behavior reconstructs as: decode the embedded executable, allocate memory for it, write it, flush the instruction cache, and transfer execution. The stack strings and ROR13 hashing exist to keep all of this invisible to a static scan of the raw blob, and capa’s Indicator Removal mapping (T1027.005) reflects the same intent.

9. Analyst Notes: Hunting Pivots

The static artifacts recovered during this analysis double as detection and hunting material. None of them depend on running the sample.

• XOR key 0x24eedcb9 — a fixed 32-bit constant bound to a decode routine. Stable across minor rebuilds and suitable for a YARA constant match or retrohunt.
• ROR13 hash 0x0726774c (LoadLibraryA) — the presence of this specific constant alongside a PEB-walking loop is characteristic of Metasploit-lineage shellcode. The hash set as a whole can seed a rule.
• VirtualAlloc→ write →NtFlushInstructionCache` → call — a behavioral sequence worth alerting on in dynamic sandboxes, independent of the specific sample.
• The two unresolved hashes — **0xc38ae110** and **0x959e0033** remain to be identified and should be carried forward as open items in any follow-on report.

10. Conclusion

Opaque shellcode does not have to be read one instruction at a time. Once capa has supplied the capability map and the addresses that matter, the analyst enters the disassembler with a plan instead of a flat blob. Binary Ninja reconstructs the obfuscation and the PEB-based resolver well enough to expose the program’s structure, and HashDB closes the gap to meaning by turning hash constants back into API names. Within a single session, the blob reads as a memory-allocating, API-hashing stager that decodes and runs an embedded executable.

The same sequence applies to almost any Windows shellcode that resolves APIs by hash, which is most of it: characterize the sample, detect its capabilities, map them to code, reconstruct the obfuscation, and resolve the hashes. The two constants that did not resolve are the useful reminder here. Triage opens a sample, it does not close it, and the honest deliverable is an annotated database plus a clear record of what is known and what is still open.

Appendix A — Indicators of Compromise

MD5     a7c3f0e6e7b6f1a0147b27a7e52088d2
SHA-1   df7205f168cb0ae912e38875823512332631bedf
SHA-256 822872c4e6799dc4c80f8863911387b31fb6f5f32c82b9b75b2b5a2886a147e4
SSDEEP  1536:UwdKlUb+Dm4s9hN1YkPDckM8HsquOBcrqqRTVrdnsqiMSXke:sI4sZ1YkPH1BcGqFVrBsr
TLSH    T16F538D13C707D47AE683407E3517BAB641393D391271E4AEFE878989A9207E176D1F0B

XOR key            0x24eedcb9
Image base         0x690000
API hash algorithm Metasploit ROR13

Appendix B — Key Addresses

0x690467, 0x690C0C   PEB Ldr module walk (API resolver)
0x690EA9             Base64 decoder
0x691355, 0x6913F8   XOR decode loops
0x693785             RC4 PRGA
0x690FBF ...         Stack-string construction (8 sites)

References

1. capa — github.com/mandiant/capa

2. capa rules — github.com/mandiant/capa-rules

3. capa import-to-bn plugin — github.com/mandiant/capa/blob/master/scripts/import-to-bn.py

4. Binary Ninja plugin guide — docs.binary.ninja/guide/plugins.html

5. OALabs HashDB — github.com/OALabs/hashdb

6. Metasploit ROR13 hashing — github.com/rapid7/metasploit-framework/blob/master/external/source/shellcode/windows/x86/src/hash.py

7. Source unpacking walkthrough — github.com/Lynk4/mare/tree/main/Malware%20Analysis/Windows/Automated%20Unpacking

8. MITRE ATT&CK — attack.mitre.org · MBC — github.com/MBCProject/mbc-markdown

The full technical analysis, sample, and extraction steps are on GitHub:

[embed]mare/Malware Analysis/Windows/Shellcode Triage and API Resolution at main · Lynk4/mare Malware Analysis and Reverse Engineering, Malware Analysis Reports.......... - mare/Malware Analysis/Windows/Shellcode…github.com

The sample analyzed here was extracted in a prior unpacking stage. To reproduce this analysis, extract the shellcode from the referenced walkthrough first. All work shown is static; the sample was not executed.

[embed]mare/Malware Analysis/Windows/Automated Unpacking at main · Lynk4/mare Malware Analysis and Reverse Engineering, Malware Analysis Reports.......... - mare/Malware Analysis/Windows/Automated…github.com