Enigma Protector 5+: Understanding Anti-Debugging, the Virtual Machine, and Finding Patching Points
logo_ru.png
Look, software security is an eternal battle of sword and shield. The internet is full of tutorials on hacking Enigma, but they all apply to older versions. But what if you're faced with a fresh Enigma 5.x or 6.x, especially on x64? "Download the tool and press a button" won't cut it. I've been through it, and now I'll tell you how it really is.
Many people think that VMProtect and Themida are great protection tools, while Enigma is supposedly for schoolchildren. This is partly true: there are indeed a ton of tools for older versions of Enigma. But the latest versions are a serious threat. The developers have improved the anti-debugging features, added a virtual machine, obfuscation, and dump protection.
In this article, I'll show you how I approached Enigma Protector x64 5.x, the pitfalls I encountered, and how I eventually found a way to bypass the protection.
---
First Encounter: Why Tools Don't Work
I took a program protected by Enigma and loaded it into Exeinfo. The analyzer showed: Enigma Protector x64 [v.5.0 - 7.0]. Well, I thought, now we'll find a video on YouTube, download the tool, and in five minutes, we'll have an unpacked EXE. Yeah, right.
All the tutorials start the same way: "Load the program into the debugger" or "Create an application dump." And here's the first problem.
· The program won't load at all in x64dbg.
· Dumpers don't work. Even if a dump is created, it's inoperable.
· dnSpy doesn't detect .NET code, even though the program is definitely on DotNet. I was only able to attach the process in x64dbg, but the process closes at the slightest attempt to move from a breakpoint. The usual: code encryption, dump protection, and debugger protection.
---
Debugger Setup: ScyllaHide and First Steps
I install the ScyllaHide and Scylla plugins. ScyllaHide helps load the program into the debugger. It finally runs and even allows stepping through the code. But the joy is short-lived: the program still closes quickly, despite ScyllaHide's settings. The anti-debugger kicks in.
But we have a small victory—the program is loaded and traced.
I move through the code, bypassing several self-decrypting sections. At one point, I come across code that looks like a wrapper for a huge, packed and encrypted section. Here's the beginning:
push rcx
push rdx
push r8
push r9
mov r8,000B5ED8E
call .00007FF7BA4D9610
mov r8.0
mov rdx,1
mov rcx,[rsp][000000018]
call .00007FF7BA4E49A0
mov r8.0
mov rdx,1
mov rcx,[rsp][000000018]
call .00007FF7BA13E400
pop r9
pop r8
pop rdx
pop rcx
call .00007FF7BA12A210
---
First dump: not .NET, but Delphi
I'm trying to dump this section using Scylla. The resulting EXE file is inoperable. And most importantly, it's not .NET, but Delphi! Enigma is written in Delphi. This doesn't make things any easier, but it does make things clearer.
I delve deeper into the code. I quickly get bogged down in the last call. Stepping through it would waste a lot of time. The anti-debugger still won't let me work calmly.
---
Exported symbols: the first clue
I look at the dumped module. It has exported symbols. Here are some of them:
0 .00007FF7BABBD06E EP_RegHardwareID
1 .00007FF7BABBD073 EP_RegHardwareIDA
2 .00007FF7BABBD078 EP_RegHardwareIDW
3 .00007FF7BABBD07D EP_RegCheckKey
4 .00007FF7BABBD082 EP_RegCheckKeyA
5 .00007FF7BABBD087 EP_RegCheckKeyW
6 .00007FF7`BABBD08C EP_RegSaveKey
...
EP stands for Enigma Protection. All of these entry points lead to useful functions.
I'm checking some of them. After a couple of long jmps, they all run into this structure:
push 005C7AC29
jmp .00007FF7`BA4F3F50
I've seen something similar in VMProtect. It's a virtual machine.
---
Finding the virtual machine interpreter
I search the unpacked code for the beginning of the interpreter:
push rsp
push rax
push rcx
push rdx
push rbx
push rbp
push rsi
push rdi
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15
pushfq
mov dl,1
mov rsi,0
lea rdi,[00007FF7`BA5032F0]
add rdi,rsi
lea rdi,[rdi]
mov rcx,1
...
I set a breakpoint at the beginning of the interpreter. The program crashes again, but now we have a call log.
---
Logging and anti-debugger detection
I'm setting up logging: at each stop in the interpreter, I'm printing RAX and the stack trace parameter. I get the log:
610 5C7C535
143336BE190 5C7CACA
143336BE1D0 5C7CACE
14335219760 5C7CAC5
630 5C7C754
7FF8DBCE0000 5C7C76D
143350D0000 5C7C778
7FF8DB160000 5C7C77F
7FF8DB17A360 5C7C770
615FF24EC83C3 5C7C777
7FF8DBD7C8C0 5C7C73F
...
I find the specific place after which the program closes. In the log, it looks like this:
...
143352119D0 5C7C549
143336BE968 5C7C544
704562405E78BE75 5C7C55E
8 5C7C56D <--- this is where the anti-debugger kicks in
I set a break condition: RAX==8 && [RSP]==0x5C7C56D. I'm looking at what happens after calling the virtual machine.
The call turned out to be part of the following construct:
mov rcx,[00007FF7BA49D9F0]
mov r8d,7
call .00007FF7BA17C750 ; virtual machine call
mov rcx,[rbp][-018]
test rcx,rcx
jnz .00007FF7BA167AA2
lea rcx,[00007FF7BA49B470]
call .00007FF7BA1270C0 ; GetModuleHandleA(ntdll.dll)
mov rcx,rax
mov rdx,rbx
call .00007FF7BA127160 ; GetProcAddress(NtSetInformationThread)
mov [rbp][-8],rax
cmp q,[rbp][-8],0
jz.00007FF7BA167ADD
call .00007FF7BA1278A0
mov rcx,rax
mov r9d,0
mov r8.0
mov edx,000000011
call q,[rbp][-8] ; call NtSetInformationThread (anti-debug)
nop
mov rcx,rbp
call .00007FF7`BA1679D0
...
NtSetInformationThread is called here—a classic anti-debugging technique. If you short-circuit the jz transition, the program won't crash.
---
How the Enigma Virtual Machine Works
The machine is mutating. The code is randomly generated, and the addresses and references in different protected modules are different. This complicates offset searching, but some blocks of code are still preserved.
Entering the interpreter:
mov [rbp][-8],ecx ; magic number on the stack
mov [rbp][-010],rdx
mov edx,[00007FF7BA5032B0] ; XOR key
mov eax,[rbp][-8]
xor eax,edx ; decode the magic number
mov [rbp][-020],eax
mov rax,[00007FF7BA5032C0] ; Relative base of the threaded code
mov edx,[rbp][-020]
mov edx,[rax][rdx]*4
mov rax,[00007FF7BA5032B8] ; Absolute base
lea rax,[rdx][rax] ; Address of the first VM instruction
mov [rbp][-000000088],rax
mov rax,[rbp][-000000088]
cmp d,[rax],000000231 ; Fetching the first instruction
jnz .00007FF7BA4ECB49
...
The address of the first instruction of the virtual machine is calculated as:
[absolute_base] + [relative_base] + (magic_number XOR key) * 4
The threaded code is always double-word aligned; the offset is specified in these double-words.
---
Exit the interpreter
mov rax,[rbp][-000000088] ; Current instruction pointer
mov rdx,[rax][020] ; Relative address of the machine code
mov rax,[00007FF7BA5032D0] ; Absolute base
lea rax,[rdx][rax] ; Machine code address
mov [rbp][-018],rax
jmps .00007FF7BA4F3E04
jmp .00007FF7`BA4ECAB8
mov rax,[rbp][-018] ; RAX = next instruction address for RET
mov rbx,[rbp][-0000000B8]
lea rsp,[rbp][0]
pop rbp
retn
Machine code calls can be made from virtual code (opcode 0x631). Upon return from a virtual code block (opcode 0x60), a return to the specified machine code block occurs.
---
Enigma Protector 5+ is a serious adversary. It has a virtual machine, anti-debugging, and mutating code. But even this protection can be bypassed.
What a researcher should do:
Use ScyllaHide to at least somehow load the program into the debugger.
Don't try to get a working dump right away—look for exported symbols.
Log virtual machine calls—this helps find points where the anti-debugger is triggered.
You don't need to completely disassemble the VM command system—knowing individual blocks is enough.
What a defender should do:
Don't rely solely on the virtual machine. Combine methods.
Update your protection—old versions break quickly.
Remember: there are no universal protections. Everything can be hacked.
logo_ru.png
Look, software security is an eternal battle of sword and shield. The internet is full of tutorials on hacking Enigma, but they all apply to older versions. But what if you're faced with a fresh Enigma 5.x or 6.x, especially on x64? "Download the tool and press a button" won't cut it. I've been through it, and now I'll tell you how it really is.
Many people think that VMProtect and Themida are great protection tools, while Enigma is supposedly for schoolchildren. This is partly true: there are indeed a ton of tools for older versions of Enigma. But the latest versions are a serious threat. The developers have improved the anti-debugging features, added a virtual machine, obfuscation, and dump protection.
In this article, I'll show you how I approached Enigma Protector x64 5.x, the pitfalls I encountered, and how I eventually found a way to bypass the protection.
---
First Encounter: Why Tools Don't Work
I took a program protected by Enigma and loaded it into Exeinfo. The analyzer showed: Enigma Protector x64 [v.5.0 - 7.0]. Well, I thought, now we'll find a video on YouTube, download the tool, and in five minutes, we'll have an unpacked EXE. Yeah, right.
All the tutorials start the same way: "Load the program into the debugger" or "Create an application dump." And here's the first problem.
· The program won't load at all in x64dbg.
· Dumpers don't work. Even if a dump is created, it's inoperable.
· dnSpy doesn't detect .NET code, even though the program is definitely on DotNet. I was only able to attach the process in x64dbg, but the process closes at the slightest attempt to move from a breakpoint. The usual: code encryption, dump protection, and debugger protection.
---
Debugger Setup: ScyllaHide and First Steps
I install the ScyllaHide and Scylla plugins. ScyllaHide helps load the program into the debugger. It finally runs and even allows stepping through the code. But the joy is short-lived: the program still closes quickly, despite ScyllaHide's settings. The anti-debugger kicks in.
But we have a small victory—the program is loaded and traced.
I move through the code, bypassing several self-decrypting sections. At one point, I come across code that looks like a wrapper for a huge, packed and encrypted section. Here's the beginning:
push rcx
push rdx
push r8
push r9
mov r8,000B5ED8E
call .00007FF7BA4D9610
mov r8.0
mov rdx,1
mov rcx,[rsp][000000018]
call .00007FF7BA4E49A0
mov r8.0
mov rdx,1
mov rcx,[rsp][000000018]
call .00007FF7BA13E400
pop r9
pop r8
pop rdx
pop rcx
call .00007FF7BA12A210
---
First dump: not .NET, but Delphi
I'm trying to dump this section using Scylla. The resulting EXE file is inoperable. And most importantly, it's not .NET, but Delphi! Enigma is written in Delphi. This doesn't make things any easier, but it does make things clearer.
I delve deeper into the code. I quickly get bogged down in the last call. Stepping through it would waste a lot of time. The anti-debugger still won't let me work calmly.
---
Exported symbols: the first clue
I look at the dumped module. It has exported symbols. Here are some of them:
0 .00007FF7BABBD06E EP_RegHardwareID
1 .00007FF7BABBD073 EP_RegHardwareIDA
2 .00007FF7BABBD078 EP_RegHardwareIDW
3 .00007FF7BABBD07D EP_RegCheckKey
4 .00007FF7BABBD082 EP_RegCheckKeyA
5 .00007FF7BABBD087 EP_RegCheckKeyW
6 .00007FF7`BABBD08C EP_RegSaveKey
...
EP stands for Enigma Protection. All of these entry points lead to useful functions.
I'm checking some of them. After a couple of long jmps, they all run into this structure:
push 005C7AC29
jmp .00007FF7`BA4F3F50
I've seen something similar in VMProtect. It's a virtual machine.
---
Finding the virtual machine interpreter
I search the unpacked code for the beginning of the interpreter:
push rsp
push rax
push rcx
push rdx
push rbx
push rbp
push rsi
push rdi
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15
pushfq
mov dl,1
mov rsi,0
lea rdi,[00007FF7`BA5032F0]
add rdi,rsi
lea rdi,[rdi]
mov rcx,1
...
I set a breakpoint at the beginning of the interpreter. The program crashes again, but now we have a call log.
---
Logging and anti-debugger detection
I'm setting up logging: at each stop in the interpreter, I'm printing RAX and the stack trace parameter. I get the log:
610 5C7C535
143336BE190 5C7CACA
143336BE1D0 5C7CACE
14335219760 5C7CAC5
630 5C7C754
7FF8DBCE0000 5C7C76D
143350D0000 5C7C778
7FF8DB160000 5C7C77F
7FF8DB17A360 5C7C770
615FF24EC83C3 5C7C777
7FF8DBD7C8C0 5C7C73F
...
I find the specific place after which the program closes. In the log, it looks like this:
...
143352119D0 5C7C549
143336BE968 5C7C544
704562405E78BE75 5C7C55E
8 5C7C56D <--- this is where the anti-debugger kicks in
I set a break condition: RAX==8 && [RSP]==0x5C7C56D. I'm looking at what happens after calling the virtual machine.
The call turned out to be part of the following construct:
mov rcx,[00007FF7BA49D9F0]
mov r8d,7
call .00007FF7BA17C750 ; virtual machine call
mov rcx,[rbp][-018]
test rcx,rcx
jnz .00007FF7BA167AA2
lea rcx,[00007FF7BA49B470]
call .00007FF7BA1270C0 ; GetModuleHandleA(ntdll.dll)
mov rcx,rax
mov rdx,rbx
call .00007FF7BA127160 ; GetProcAddress(NtSetInformationThread)
mov [rbp][-8],rax
cmp q,[rbp][-8],0
jz.00007FF7BA167ADD
call .00007FF7BA1278A0
mov rcx,rax
mov r9d,0
mov r8.0
mov edx,000000011
call q,[rbp][-8] ; call NtSetInformationThread (anti-debug)
nop
mov rcx,rbp
call .00007FF7`BA1679D0
...
NtSetInformationThread is called here—a classic anti-debugging technique. If you short-circuit the jz transition, the program won't crash.
---
How the Enigma Virtual Machine Works
The machine is mutating. The code is randomly generated, and the addresses and references in different protected modules are different. This complicates offset searching, but some blocks of code are still preserved.
Entering the interpreter:
mov [rbp][-8],ecx ; magic number on the stack
mov [rbp][-010],rdx
mov edx,[00007FF7BA5032B0] ; XOR key
mov eax,[rbp][-8]
xor eax,edx ; decode the magic number
mov [rbp][-020],eax
mov rax,[00007FF7BA5032C0] ; Relative base of the threaded code
mov edx,[rbp][-020]
mov edx,[rax][rdx]*4
mov rax,[00007FF7BA5032B8] ; Absolute base
lea rax,[rdx][rax] ; Address of the first VM instruction
mov [rbp][-000000088],rax
mov rax,[rbp][-000000088]
cmp d,[rax],000000231 ; Fetching the first instruction
jnz .00007FF7BA4ECB49
...
The address of the first instruction of the virtual machine is calculated as:
[absolute_base] + [relative_base] + (magic_number XOR key) * 4
The threaded code is always double-word aligned; the offset is specified in these double-words.
---
Exit the interpreter
mov rax,[rbp][-000000088] ; Current instruction pointer
mov rdx,[rax][020] ; Relative address of the machine code
mov rax,[00007FF7BA5032D0] ; Absolute base
lea rax,[rdx][rax] ; Machine code address
mov [rbp][-018],rax
jmps .00007FF7BA4F3E04
jmp .00007FF7`BA4ECAB8
mov rax,[rbp][-018] ; RAX = next instruction address for RET
mov rbx,[rbp][-0000000B8]
lea rsp,[rbp][0]
pop rbp
retn
Machine code calls can be made from virtual code (opcode 0x631). Upon return from a virtual code block (opcode 0x60), a return to the specified machine code block occurs.
---
Enigma Protector 5+ is a serious adversary. It has a virtual machine, anti-debugging, and mutating code. But even this protection can be bypassed.
What a researcher should do:
Use ScyllaHide to at least somehow load the program into the debugger.
Don't try to get a working dump right away—look for exported symbols.
Log virtual machine calls—this helps find points where the anti-debugger is triggered.
You don't need to completely disassemble the VM command system—knowing individual blocks is enough.
What a defender should do:
Don't rely solely on the virtual machine. Combine methods.
Update your protection—old versions break quickly.
Remember: there are no universal protections. Everything can be hacked.
