Anti-Analysis Techniques
Overview
Malware uses anti-analysis techniques to detect and evade analysis environments — debuggers, virtual machines, sandboxes, and automated analysis systems. Understanding these techniques is essential for bypassing them during reverse engineering. This file covers common anti-debugging, anti-VM, anti-sandbox, and anti-disassembly techniques along with their bypasses.
Anti-Debugging Techniques
Windows Anti-Debug
IsDebuggerPresent
Checks PEB.BeingDebugged flag
Bypass: Set PEB.BeingDebugged = 0, or hook return value to 0
CheckRemoteDebuggerPresent
Queries NtQueryInformationProcess with ProcessDebugPort
Bypass: Hook and return FALSE
NtQueryInformationProcess
ProcessDebugPort (0x07) — returns non-zero if debugged
ProcessDebugObjectHandle (0x1E) — handle exists if debugged
ProcessDebugFlags (0x1F) — returns 0 if debugged
Bypass: Hook ntdll and modify output values
PEB flags
PEB.BeingDebugged (offset 0x02)
PEB.NtGlobalFlag (offset 0x68/0xBC) — set to 0x70 when process is created by a debugger
PEB.ProcessHeap.Flags / ForceFlags — different under debugger
Bypass: Manually zero these fields in memory
OutputDebugStringA
Void function — call SetLastError(value), then OutputDebugStringA(), then GetLastError()
Under debugger: GetLastError() returns 0 (debugger consumed the error)
Not debugged: GetLastError() returns original value
Bypass: Hook GetLastError to return the expected non-zero value
CloseHandle with invalid handle
Raises EXCEPTION_INVALID_HANDLE under debugger
Bypass: Handle the exception in debugger settings
Linux Anti-Debug
# Common Linux anti-debug checks:
# 1. ptrace(PTRACE_TRACEME) — fails if already being traced
# Returns -1 under debugger
# Bypass: catch syscall ptrace, set $rax = 0
# 2. /proc/self/status — TracerPid field
# Non-zero TracerPid = being debugged
# Bypass: Hook open/read for this file, return TracerPid: 0
# 3. /proc/self/exe readlink
# May show debugger path
# Bypass: Modify GDB argv[0] or use symlink
# 4. SIGTRAP handler
# Program installs SIGTRAP handler, then INT 3
# Under debugger, INT 3 is caught by debugger instead
# Bypass: Pass SIGTRAP to the program (handle SIGTRAP nostop pass)
# 5. Timing checks
# Measure time between operations; debugger introduces delay
# Bypass: Patch timing comparison or modify return values
Timing-Based Detection
Techniques:
GetTickCount / GetTickCount64
Compare values before and after code block
Large delta = debugger stepping
RDTSC (Read Time-Stamp Counter)
Assembly instruction reading CPU cycle counter
Significant delta = single-stepping detected
QueryPerformanceCounter
High-resolution timer comparison
NtQuerySystemTime / GetSystemTimeAsFileTime
System time comparison
Bypass:
- Patch the comparison jump (JA/JB → JMP or NOP)
- Modify the timer return value at breakpoint
- In x64dbg: use ScyllaHide to handle timing checks
- In GDB: set a breakpoint on the comparison, modify flags
Anti-VM Techniques
Hardware Detection
Registry checks (Windows):
HKLM\SYSTEM\CurrentControlSet\Services\VBoxGuest
HKLM\SYSTEM\CurrentControlSet\Services\VMTools
HKLM\SOFTWARE\VMware, Inc.\VMware Tools
HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions
Process checks:
vmtoolsd.exe, vmwaretray.exe (VMware)
VBoxService.exe, VBoxTray.exe (VirtualBox)
qemu-ga (QEMU)
File checks:
C:\Windows\System32\drivers\vmmouse.sys (VMware)
C:\Windows\System32\drivers\VBoxMouse.sys (VirtualBox)
MAC address OUI:
00:0C:29:xx:xx:xx (VMware)
08:00:27:xx:xx:xx (VirtualBox)
00:16:3E:xx:xx:xx (Xen)
52:54:00:xx:xx:xx (QEMU/KVM)
Hardware strings:
BIOS vendor: "innotek GmbH" (VirtualBox), "Phoenix" (VMware)
System manufacturer: "VMware, Inc."
Disk model: "VBOX HARDDISK", "VMware Virtual"
CPUID-Based Detection
The CPUID instruction returns processor information.
Hypervisor detection:
CPUID with EAX=1 → bit 31 of ECX is the hypervisor present bit
If set, a hypervisor is running
Hypervisor brand string:
CPUID with EAX=0x40000000 → returns hypervisor vendor string
"VMwareVMware" — VMware
"VBoxVBoxVBox" — VirtualBox
"Microsoft Hv" — Hyper-V
"KVMKVMKVM" — KVM
"XenVMMXenVMM" — Xen
Bypass: Modify CPUID results via hypervisor settings
VirtualBox: VBoxManage modifyvm <name> --paravirtprovider none
VMware: monitor_control.restrict_backdoor = "TRUE" in .vmx
Linux VM Detection
# Common checks malware uses on Linux:
# 1. dmidecode (requires root)
# Contains "VirtualBox", "VMware", "QEMU" etc.
# 2. /sys/class/dmi/id/product_name
# "VirtualBox", "VMware Virtual Platform"
# 3. lspci output
# VirtualBox / VMware PCI devices
# 4. CPU model
# /proc/cpuinfo may show "QEMU Virtual CPU"
# 5. Disk device names
# /dev/vda (virtio), /dev/xvda (Xen)
VM Evasion Bypasses
| Technique | Bypass |
|---|---|
| Registry checks | Delete or rename VM-related registry keys |
| Process checks | Rename or stop VM guest tools services |
| File checks | Delete or rename VM driver files |
| MAC address | Change MAC address to a non-VM OUI |
| CPUID | Use hypervisor settings to hide brand string |
| Hardware strings | Modify BIOS/DMI strings via VM settings |
| ACPI tables | Some VMs allow custom ACPI table names |
Anti-Sandbox Techniques
Environment Checks
Malware checks for signs of a sandbox or automated analysis:
Low resource detection:
- RAM < 2 GB
- Disk < 60 GB
- CPU cores < 2
- Screen resolution too low (800x600)
User activity detection:
- No recent documents
- No browser history
- No installed applications beyond defaults
- Mouse movement patterns (sandboxes don't move the mouse)
- Uptime too short (recently booted VM)
Username/hostname checks:
- "malware", "sandbox", "virus", "sample", "test"
- "John Doe" or other default names
- Generic hostnames like "DESKTOP-XXXXXXX"
Time-Based Evasion
Sleep-based evasion:
Malware calls Sleep() for extended periods (e.g., 10+ minutes)
Sandbox timeout occurs before malware activates
Bypass: Patch Sleep() to return immediately, or hook and reduce
the sleep duration
Time bomb:
Malware only activates after a specific date/time
Bypass: Set system clock to a future date
Gradual activation:
Malware performs benign activity first, then activates payload
after N iterations or N minutes
Bypass: Extend sandbox execution time, patch loop counters
Network Checks
Internet connectivity:
Malware checks for real internet by contacting known-good sites
(e.g., google.com, microsoft.com)
Bypass: Use INetSim to simulate responses
IP reputation:
Malware queries its external IP and checks against known
sandbox/VPN/cloud IP ranges
Bypass: Route traffic through a residential IP or VPN
Domain resolution:
Some sandboxes resolve all domains to the same IP (sinkhole)
Malware queries a non-existent domain; if it resolves, it's a sandbox
Bypass: Configure DNS to return NXDOMAIN for invalid domains
Anti-Disassembly Techniques
Opaque Predicates
Conditional jumps with predetermined outcomes that confuse disassemblers:
xor eax, eax ; EAX = 0
jz real_code ; Always taken (ZF always set)
db 0xE8 ; Fake CALL instruction (confuses linear disassembly)
real_code:
; actual code continues here
Bypass: Disassemblers like Ghidra and IDA handle most opaque predicates
through flow analysis. Manual patching may be needed for complex cases.
Junk Byte Insertion
Insert invalid or misleading bytes between real instructions:
jmp skip_junk
db 0x90, 0x90, 0xCC, 0xFF ; junk bytes
skip_junk:
; real code
The disassembler may try to interpret junk bytes as instructions,
misaligning the disassembly.
Bypass: Start disassembly from the correct offset (right-click →
"Disassemble" at the jump target in Ghidra/IDA).
Control Flow Flattening
Replace normal control flow (if/else, loops) with a dispatcher:
switch(state) {
case 0: do_init(); state = 3; break;
case 1: do_network(); state = 4; break;
case 2: do_cleanup(); state = 5; break;
case 3: do_check(); state = 1; break;
...
}
All basic blocks jump back to a central dispatcher, hiding the
real control flow.
Bypass: Trace execution to determine actual block ordering,
then reconstruct the original flow.
Dynamic Code Generation
Malware generates code at runtime instead of containing it statically:
1. Allocate executable memory (VirtualAlloc with PAGE_EXECUTE_READWRITE)
2. Decrypt or decompress shellcode into the allocated memory
3. Jump to the generated code
The generated code does not appear in static disassembly.
Bypass: Set breakpoint on VirtualAlloc, dump the generated code
after it's written, then disassemble the dump.