Understanding XOR Encryption + Jigsaw Obfuscation

In this research, I used a two-layer obfuscation approach: XOR Encryption followed by Jigsaw Obfuscation. Both techniques are simple yet powerful when combined.

XOR Encryption Layer XOR (Exclusive OR) is a fundamental bitwise operation used widely in cryptography and malware development because it is fast, reversible, and requires no complex libraries. How XOR Works (Simple Analogy): Imagine each byte in the shellcode is a light switch. XORing it with a key is like flipping the switch according to a secret rule. Doing it again with the same key brings it back to the original state. Mathematical Representation: For every byte bi b_i bi​ in the original shellcode:

Encryptedi=bi⊕K

Where:

• ⊕ \oplus ⊕ = Bitwise XOR operation
• K K K = Secret key (I used 0xAA) Decryption is exactly the same operation:

Originali=Encryptedi⊕K

Why XOR is effective for obfuscation:

• It completely changes the byte values, breaking signature-based detection.
• It has very low computational cost.
• When used with a good key, it removes obvious shellcode patterns (like 0xFC 0x48).

However, XOR alone can sometimes produce high entropy (random-looking) data, which some AVs flag. This is where the second layer helps. Jigsaw Obfuscation (Byte Permutation):

Jigsaw is not encryption — it is a permutation technique. Instead of changing the value of bytes, it changes their order.

Mathematical Concept:

Let the encrypted shellcode be an array E E E of length n n n.

1. Create a position array: P=[0,1,2,…,n−1] P = [0, 1, 2, …, n-1] P=[0,1,2,…,n−1]
2. Randomly shuffle P P P (this becomes our secret map).
3. Build the Jigsaw array:

J[i]=E[P[i]]for all i

At runtime, reconstruction is done by:

Recovered[P[i]]=J[i]

This is basically solving a puzzle using a lookup table.

Detailed Analogy: Imagine your original shellcode is a beautiful completed picture made of 1000 puzzle pieces neatly arranged on a table.

This picture (the shellcode) contains dangerous instructions for example, “connect back to my IP and give me a shell”. Antivirus tools have seen this exact picture many times, so they can easily recognize and block it.

What the Jigsaw Technique Does:

Step 1: Cut the Picture into Pieces We break the completed picture into 1000 individual small pieces. Each piece is one byte of your shellcode.

Step 2: Mix Them Up Randomly (Shuffling) Now, instead of keeping the pieces in correct order, we throw them all over the table in random order.

So now you have a big pile of 1000 mixed pieces. Even if someone looks at this pile, they cannot understand what the full picture is supposed to be.

Step 3: Create a Secret Map While mixing the pieces, we create a secret instruction paper (called positions[] array).

This map says things like:

• Piece number 65 belongs to position 0
• Piece number 18 belongs to position 1
• Piece number 36 belongs to position 2
• … and so on for all 1000 pieces.

This secret map is the most important part.

Why Combining XOR + Jigsaw is Powerful

• XOR changes the values (makes it look unfamiliar).
• Jigsaw changes the order (breaks sequential patterns and signatures).
• Together they create multi-layered obfuscation with low entropy (because Jigsaw avoids making data look too random).

This combination makes static analysis much harder because:

• On disk, the binary contains only shuffled + encrypted bytes.
• No obvious malicious signatures are present.
• The real shellcode only appears in memory after reconstruction and decryption.

Successful AV Bypass

As part of this research, I tested my custom-built shellcode loader against real antivirus solutions.

Key Achievement:

By writing my own evasion code from scratch (custom XOR + Jigsaw implementation), I was able to successfully bypass static detection of AV engines.

What I Observed:

• The final compiled loader binary showed 0 detections on VirusTotal for static analysis.
• Windows Defender did not flag the binary as malicious when downloaded or executed in a test environment.
• The reverse shell executed successfully without triggering real-time protection.

This bypass was possible because:

• The shellcode never existed in its original form on disk.
• Only encrypted and shuffled data was present in the executable.
• All reconstruction and decryption happened dynamically in memory.

This proves that even with simple, custom-written techniques like XOR Encryption + Jigsaw Permutation, it is possible to bypass modern static-based detection mechanisms when implemented carefully.

Important Note: This research was conducted only in an isolated virtual lab for educational and defensive research purposes. Modern EDR solutions (with behavioral analysis) may still detect the runtime behavior (RWX memory allocation + execution). This work is meant to understand attack techniques so we can build better defenses.

Happy Hacking…!