Memory-Based Malware Hunting

Overview

Memory-based malware hunting uses memory forensics techniques to detect malicious code that may not be visible on disk. Fileless malware, process injection, DLL hollowing, rootkits, and packed/encrypted payloads all leave traces in memory that can be detected with Volatility plugins and YARA rules. This file covers the key detection techniques for identifying malware artifacts in memory dumps.

Detecting Injected Code with Malfind

Malfind identifies VAD (Virtual Address Descriptor) regions with suspicious characteristics — primarily memory regions that are both writable and executable (PAGE_EXECUTE_READWRITE) and contain executable code.

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Scan all processes for injected code
vol -f memory.raw windows.malware.malfind

# Scan a specific process
vol -f memory.raw windows.malware.malfind --pid 1234

# Malfind output includes:
#   Process name and PID
#   VAD start address and protection flags
#   First 64 bytes (hex dump)
#   Disassembly of the first bytes

# Dump injected regions for further analysis
vol -f memory.raw -o /output/ windows.malware.malfind --dump

Interpreting Malfind Results:

Indicator Meaning
PAGE_EXECUTE_READWRITE Memory is writable + executable (unusual for legitimate code)
MZ header (4D 5A) PE file mapped in memory (injected DLL or shellcode loader)
Shellcode patterns (FC E8, EB FE) Common shellcode prologues
No associated module Code not backed by a file on disk
Process = svchost, explorer, etc. Injection into trusted system processes

Detecting Process Hollowing

Process hollowing (RunPE) creates a legitimate process in a suspended state, replaces its memory with malicious code, then resumes execution. The process appears legitimate in task managers but runs attacker code.

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Detect hollowed processes
vol -f memory.raw windows.malware.hollowprocesses

# Manual detection approach:
# 1. Compare the process image path (from PEB) with the actual image in memory
vol -f memory.raw windows.dlllist --pid 1234
# The first entry is the main executable — check if its base address
# matches the expected image

# 2. Dump the process executable and compare with the on-disk version
vol -f memory.raw -o /output/ windows.dumpfiles --pid 1234
# Hash both and compare — mismatch indicates hollowing

Hollowing Detection Indicators:

Indicator Description
PEB image path ≠ in-memory image Process claims to be one binary but runs another
Section permissions mismatch .text section is writable (should be RX only)
Image base mismatch Base address doesn't match PE header's ImageBase
Missing PE sections Expected sections from the original binary are absent

Detecting Hidden DLLs

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# LdrModules compares three different DLL lists maintained by Windows
vol -f memory.raw windows.malware.ldrmodules

# A DLL that is False in all three loader lists but exists in the VAD
# tree is likely manually mapped (injected without LoadLibrary)

# Check the IAT (Import Address Table) for hooking
vol -f memory.raw windows.iat --pid 1234
# Hooked imports will point to unexpected addresses

Rootkit Detection

Kernel-Level Hooks

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Check SSDT (System Service Descriptor Table) for hooks
vol -f memory.raw windows.ssdt
# Hooked entries will point outside ntoskrnl.exe or win32k.sys

# Check for driver IRP hooks
vol -f memory.raw windows.driverirp
# IRP handlers pointing to suspicious driver addresses

# Scan for kernel modules
vol -f memory.raw windows.driverscan

# Compare loaded modules list with scanned modules
vol -f memory.raw windows.modules
vol -f memory.raw windows.modscan
# Modules found by modscan but not in modules list = potentially hidden

Linux Rootkit Detection

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Check for hidden kernel modules
vol -f memory.lime linux.hidden_modules

# Check system call table for hooks
vol -f memory.lime linux.check_syscall
# Hooks pointing outside the kernel text range = rootkit

# List loaded kernel modules
vol -f memory.lime linux.lsmod

# Check for eBPF-based rootkits
vol -f memory.lime linux.ebpf

YARA Scanning in Memory

YARA rules enable custom pattern matching across entire memory dumps to detect known malware signatures, shellcode patterns, or custom indicators.

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Scan process VADs with YARA rules
vol -f memory.raw windows.vadyarascan --yara-file /path/to/rules.yar

# Scan specific process
vol -f memory.raw windows.vadyarascan --yara-file /path/to/rules.yar --pid 1234

Example YARA Rules for Memory Scanning:

rule Cobalt_Strike_Beacon {
    meta:
        description = "Detect Cobalt Strike Beacon in memory"
    strings:
        $pipe = "\\\\.\\pipe\\msagent_" ascii
        $config = { 00 01 00 01 00 02 ?? ?? 00 02 00 01 00 02 ?? ?? }
        $sleep_mask = { 4C 8B 53 08 45 8B 0A 45 8B 5A 04 4D 8D 52 08 }
    condition:
        any of them
}

rule Meterpreter_Reverse_TCP {
    meta:
        description = "Detect Meterpreter reverse TCP shellcode"
    strings:
        $api_hash = { 72 6E 69 74 }  // "rnit" from ws2_32.dll
        $stage = "metsrv.dll" ascii wide
    condition:
        any of them
}

rule Generic_Shellcode {
    meta:
        description = "Detect common shellcode patterns"
    strings:
        $win_api_resolve = { 64 A1 30 00 00 00 }  // mov eax, fs:[0x30] (PEB access)
        $syscall_stub = { 0F 05 }                   // syscall instruction
        $call_pop = { E8 00 00 00 00 (58 | 59 | 5A | 5B | 5D | 5E | 5F) }
    condition:
        any of them
}

Credential Extraction

Malware and attackers frequently target credentials stored in memory.

# Volatility 3
# https://github.com/volatilityfoundation/volatility3

# Dump password hashes from the SAM
vol -f memory.raw windows.registry.hashdump

# Dump cached domain credentials
vol -f memory.raw windows.registry.cachedump

# Dump LSA secrets
vol -f memory.raw windows.registry.lsadump

# These plugins extract the same data that Mimikatz targets
# Useful for understanding what credentials an attacker could have accessed

Malware Hunting Workflow

Memory dump received
    │
    ├── 1. Identify OS and validate dump
    │   └── vol -f dump.raw windows.info
    │
    ├── 2. Process enumeration
    │   ├── vol windows.pslist → baseline process list
    │   ├── vol windows.pstree → check parent-child relationships
    │   ├── vol windows.psscan → find hidden/terminated processes
    │   └── vol windows.malware.psxview → cross-reference all methods
    │
    ├── 3. Malware detection
    │   ├── vol windows.malware.malfind → injected code
    │   ├── vol windows.malware.hollowprocesses → process hollowing
    │   ├── vol windows.malware.ldrmodules → hidden DLLs
    │   └── vol windows.vadyarascan → YARA signature scan
    │
    ├── 4. Persistence and hooks
    │   ├── vol windows.ssdt → system call hooks
    │   ├── vol windows.svcscan → malicious services
    │   ├── vol windows.registry.printkey → Run keys
    │   └── vol windows.scheduled_tasks → scheduled tasks
    │
    ├── 5. Network indicators
    │   └── vol windows.netscan → C2 connections, beacons
    │
    ├── 6. Extract artifacts
    │   ├── vol windows.dumpfiles → suspect executables
    │   ├── vol windows.registry.hashdump → compromised credentials
    │   └── Dump malfind regions for reverse engineering
    │
    └── 7. Document IOCs and correlate with disk/network evidence

References

Tools

Further Reading