Debugging Techniques
Overview
Debugging malware involves stepping through code execution in real-time to observe register values, memory contents, and control flow at specific points. Debuggers allow setting breakpoints, examining stack frames, and modifying execution — essential for understanding unpacking routines, encrypted configurations, C2 protocols, and anti-analysis bypasses.
GDB (GNU Debugger)
GDB is the standard debugger for Linux binaries.
# GDB
# https://www.sourceware.org/gdb/
# Debug an executable
gdb ./sample
# Debug with arguments
gdb --args ./sample arg1 arg2
# Attach to a running process
gdb -p <pid>
# Load a core dump
gdb ./sample core
# Execute GDB commands from command line
gdb -ex 'break main' -ex 'run' ./sample
# Quiet mode (suppress startup messages)
gdb -q ./sample
Essential GDB Commands
# GDB
# https://www.sourceware.org/gdb/
# --- Execution Control ---
run # Start program execution
run arg1 arg2 # Start with arguments
continue (c) # Continue after breakpoint
step (s) # Step into (follow calls)
next (n) # Step over (skip calls)
stepi (si) # Step one instruction
nexti (ni) # Step one instruction (skip calls)
finish # Run until current function returns
kill # Kill the running program
# --- Breakpoints ---
break main # Break at function
break *0x401000 # Break at address
break *0x401000 if $rax == 0 # Conditional breakpoint
info breakpoints # List all breakpoints
delete 1 # Delete breakpoint #1
disable 1 # Disable breakpoint #1
enable 1 # Enable breakpoint #1
# --- Registers ---
info registers # Show all registers
print $rax # Print specific register
print/x $rax # Print in hex
set $rax = 0 # Modify register value
# --- Memory ---
x/10x $rsp # Examine 10 hex words at RSP
x/20i $rip # Disassemble 20 instructions at RIP
x/s 0x402000 # Print string at address
x/10b 0x402000 # Print 10 bytes at address
x/10wx $rsp # Print 10 32-bit words at RSP
# --- Stack ---
backtrace (bt) # Show call stack
frame 0 # Select stack frame
info locals # Show local variables
info args # Show function arguments
# --- Information ---
info proc mappings # Show memory map
info sharedlibrary # Show loaded libraries
info threads # Show threads
info functions # List all functions
disassemble main # Disassemble a function
GDB for Malware Analysis
# GDB
# https://www.sourceware.org/gdb/
# Break on common malware activities
# Break on network connections
break connect
break socket
break send
break recv
# Break on file operations
break open
break fopen
break write
break unlink
# Break on process execution
break execve
break system
break fork
# Break on memory operations (unpacking)
break mmap
break mprotect
# Break on dynamic library loading
break dlopen
break dlsym
# Dump memory region to file
dump binary memory dump.bin 0x400000 0x410000
# Watch a memory address for changes
watch *0x602010
# Catch system calls
catch syscall connect
catch syscall execve
GDB Enhanced with GEF
GEF (GDB Enhanced Features) provides a better interface for reverse engineering and exploitation.
# GEF
# https://github.com/hugsy/gef
# Install GEF
bash -c "$(curl -fsSL https://gef.blah.cat/sh)"
# GEF-specific commands:
# context — show registers, stack, code, threads
# vmmap — show memory mappings
# heap — heap analysis commands
# checksec — check binary security features
# pattern create — create de Bruijn pattern
# xinfo addr — detailed info about an address
# search-pattern — search memory for a pattern
# got — show GOT entries
x64dbg (Windows)
x64dbg is the primary open-source user-mode debugger for Windows malware analysis. It is a GUI debugger supporting both 32-bit (x32dbg) and 64-bit (x64dbg) binaries.
Key x64dbg Operations
Navigation:
Ctrl+G — Go to address/expression
Ctrl+F — Find pattern in current module
Ctrl+B — Binary search
Space — Assemble instruction (patch)
Execution:
F9 — Run
F7 — Step into
F8 — Step over
Ctrl+F9 — Run until return
F2 — Toggle breakpoint
Shift+F2 — Conditional breakpoint
Breakpoints:
F2 on line — Software breakpoint
Hardware BP — Breakpoints → Hardware breakpoint
Memory BP — Right-click memory → Set memory breakpoint
Conditional — Breakpoint → Edit → set condition
Patching:
Space — Edit instruction
Ctrl+P — Patches window
File → Patch — Save patched binary
x64dbg Malware Analysis Workflow
1. Load sample: File → Open → select malware
2. Set initial breakpoints:
- bp VirtualAlloc (memory allocation for unpacking)
- bp VirtualProtect (change page permissions)
- bp CreateFileW (file operations)
- bp RegSetValueExW (registry modifications)
- bp CreateRemoteThread (process injection)
- bp IsDebuggerPresent (anti-debug check)
3. Run (F9) and observe breakpoint hits
4. At VirtualAlloc breaks:
- Note the allocated address (return value in EAX/RAX)
- Set memory breakpoint on that region
5. At VirtualProtect breaks:
- Check if making memory executable (PAGE_EXECUTE_READWRITE = 0x40)
- The protected region likely contains unpacked code
6. Follow execution to find the OEP (Original Entry Point)
7. Dump the unpacked module
x64dbg Scripting
// x64dbg script — trace API calls
bp CreateFileW
bp RegSetValueExW
bp connect
bp InternetOpenA
// Log each breakpoint hit
bpcnd CreateFileW, "log \"CreateFileW: {s:arg.get(0)}\"; run"
bpcnd RegSetValueExW, "log \"RegSetValueExW: {s:arg.get(1)}\"; run"
Debugging Anti-Debug Protected Malware
Common Anti-Debug Techniques and Bypasses
| Technique | Bypass |
|---|---|
IsDebuggerPresent |
Set breakpoint, change return value to 0 |
CheckRemoteDebuggerPresent |
Hook and force FALSE return |
NtQueryInformationProcess (ProcessDebugPort) |
Hook and return 0 |
| PEB.BeingDebugged flag | Manually set PEB.BeingDebugged = 0 |
| Timing checks (GetTickCount, RDTSC) | Patch jump or modify return value |
| INT 2D / INT 3 | Skip over or NOP out |
| Exception-based detection | Pass exceptions to the program |
GDB Anti-Debug Bypass
# GDB
# https://www.sourceware.org/gdb/
# Bypass ptrace anti-debug (PTRACE_TRACEME check)
# Malware calls ptrace(PTRACE_TRACEME) — returns -1 if already traced
catch syscall ptrace
# When caught, set return value to 0:
# set $rax = 0
# continue
x64dbg Anti-Debug Bypass
In x64dbg:
1. Plugins → ScyllaHide (if installed)
- Hides debugger from common detection methods
2. Manual bypass:
- bp IsDebuggerPresent
- When hit: set EAX = 0, then run
- bp NtQueryInformationProcess
- When hit: modify output buffer to hide debugger
3. PEB bypass:
- In dump window, go to PEB address
- Set BeingDebugged byte to 0
- Set NtGlobalFlag to 0
Breakpoint Strategies
API Breakpoints for Malware Analysis
Unpacking:
VirtualAlloc / VirtualProtect / NtAllocateVirtualMemory
Networking:
socket / connect / send / recv / WSAStartup
InternetOpenA / HttpSendRequestA / URLDownloadToFileA
File operations:
CreateFileA/W / ReadFile / WriteFile / DeleteFileA/W
Registry:
RegOpenKeyExA/W / RegSetValueExA/W / RegCreateKeyExA/W
Process injection:
OpenProcess / VirtualAllocEx / WriteProcessMemory / CreateRemoteThread
NtUnmapViewOfSection (process hollowing)
Crypto:
CryptEncrypt / CryptDecrypt / CryptAcquireContextA
BCryptEncrypt / BCryptDecrypt