API Call Monitoring

Overview

API call monitoring traces the system calls and library functions a malware sample invokes at runtime. This reveals the sample's exact interactions with the operating system — file operations, network connections, process manipulation, registry access, and more. Tracing provides a detailed, ordered log of every action the malware takes.

Linux System Call Tracing

strace

strace intercepts and records system calls made by a process and the signals it receives.

# strace
# https://github.com/strace/strace

# Trace all system calls of a program
strace ./sample

# Trace with timestamps
strace -t ./sample

# Trace with relative timestamps (time between calls)
strace -r ./sample

# Trace with microsecond timestamps
strace -tt ./sample

# Save output to a file
strace -o trace.log ./sample

# Trace specific system call categories
strace -e trace=network ./sample    # Network calls only
strace -e trace=file ./sample       # File operations only
strace -e trace=process ./sample    # Process management only
strace -e trace=memory ./sample     # Memory management only
strace -e trace=signal ./sample     # Signal handling only

# Trace specific system calls
strace -e trace=open,read,write,connect,socket ./sample

# Follow child processes (fork/clone)
strace -f ./sample

# Follow forks with separate output files
strace -ff -o trace ./sample
# Creates trace.<pid> for each process

# Attach to a running process
strace -p <pid>

# Show string arguments up to N characters (default: 32)
strace -s 256 ./sample

# Count time and calls per syscall (summary)
strace -c ./sample

# Count time per call with detailed output
strace -C ./sample

# Print non-ASCII string arguments in hex notation
strace -x ./sample

Key System Calls to Monitor

System Call Purpose
open / openat Open files
read / write Read/write file descriptors
connect Establish network connection
socket Create network socket
bind / listen / accept Server-side networking
sendto / recvfrom Send/receive data (UDP)
execve Execute a program
fork / clone Create new process/thread
mmap / mprotect Map memory / change permissions
unlink Delete a file
ptrace Debug/trace another process (anti-debug)
kill Send signal to a process
dup2 Redirect file descriptors (reverse shells)

strace Analysis Examples

# strace
# https://github.com/strace/strace

# Find network connections
strace -e trace=network -s 256 ./sample 2>&1 | grep connect

# Find file access
strace -e trace=file -s 256 ./sample 2>&1 | grep -E 'open|unlink|rename'

# Find process execution
strace -e trace=process ./sample 2>&1 | grep execve

# Find DNS resolution
strace -e trace=network ./sample 2>&1 | grep -A2 'connect.*53'

# Detect reverse shell pattern (dup2 + execve /bin/sh)
strace -f -e trace=dup2,execve ./sample 2>&1

Linux Library Call Tracing

ltrace

ltrace intercepts dynamic library calls (libc, libssl, etc.) and signals.

# ltrace
# https://www.ltrace.org/

# Trace library calls
ltrace ./sample

# Trace with timestamps
ltrace -t ./sample

# Save output to file
ltrace -o ltrace.log ./sample

# Trace specific library calls
ltrace -e malloc+free+strlen+strcmp ./sample

# Follow child processes
ltrace -f ./sample

# Show string arguments up to N characters
ltrace -s 256 ./sample

# Trace calls from a specific library only
ltrace -l libc.so.6 ./sample

# Trace system calls as well as library calls
ltrace -S ./sample

# Count calls (summary)
ltrace -c ./sample

# Attach to a running process
ltrace -p <pid>

# Show time spent in each call
ltrace -T ./sample

Key Library Calls to Monitor

Library Call Purpose
system() Execute shell command
popen() Execute command and open pipe
dlopen() / dlsym() Load shared library at runtime
fopen() / fwrite() File I/O
getenv() / setenv() Environment variable access
strcmp() / strstr() String comparison (password checks, config parsing)
malloc() / free() Memory allocation patterns
SSL_connect() TLS/SSL connection (encrypted C2)
gethostbyname() DNS resolution
crypt() Password hashing

Dynamic Instrumentation with Frida

Frida allows hooking arbitrary functions at runtime without modifying the binary.

# Frida
# https://github.com/frida/frida

# Attach to a running process by name
frida -n sample_process -l hook_script.js

# Spawn and trace a new process
frida -f ./sample -l hook_script.js

# List running processes
frida-ps

Frida Hook Script for Linux

// Frida
// https://github.com/frida/frida

// Hook connect() to monitor outbound connections
Interceptor.attach(Module.findExportByName(null, "connect"), {
    onEnter: function(args) {
        var sockaddr = args[1];
        var family = sockaddr.readU16();
        if (family === 2) {  // AF_INET
            var port = (sockaddr.add(2).readU8() << 8) | sockaddr.add(3).readU8();
            var ip = sockaddr.add(4).readU8() + "." +
                     sockaddr.add(5).readU8() + "." +
                     sockaddr.add(6).readU8() + "." +
                     sockaddr.add(7).readU8();
            console.log("[connect] " + ip + ":" + port);
        }
    }
});

// Hook open() to monitor file access
Interceptor.attach(Module.findExportByName(null, "open"), {
    onEnter: function(args) {
        console.log("[open] " + args[0].readUtf8String());
    }
});

// Hook execve() to monitor process execution
Interceptor.attach(Module.findExportByName(null, "execve"), {
    onEnter: function(args) {
        console.log("[execve] " + args[0].readUtf8String());
    }
});

// Hook system() to monitor shell commands
Interceptor.attach(Module.findExportByName(null, "system"), {
    onEnter: function(args) {
        console.log("[system] " + args[0].readUtf8String());
    }
});

Frida Hook Script for Windows

// Frida
// https://github.com/frida/frida

// Hook CreateFileW to monitor file access
var CreateFileW = Module.findExportByName("kernel32.dll", "CreateFileW");
Interceptor.attach(CreateFileW, {
    onEnter: function(args) {
        console.log("[CreateFileW] " + args[0].readUtf16String());
    }
});

// Hook RegSetValueExW to monitor registry writes
var RegSetValueExW = Module.findExportByName("advapi32.dll", "RegSetValueExW");
Interceptor.attach(RegSetValueExW, {
    onEnter: function(args) {
        console.log("[RegSetValueExW] Value: " + args[1].readUtf16String());
    }
});

// Hook CreateProcessW to monitor process creation
var CreateProcessW = Module.findExportByName("kernel32.dll", "CreateProcessW");
Interceptor.attach(CreateProcessW, {
    onEnter: function(args) {
        if (!args[0].isNull()) {
            console.log("[CreateProcessW] App: " + args[0].readUtf16String());
        }
        if (!args[1].isNull()) {
            console.log("[CreateProcessW] Cmd: " + args[1].readUtf16String());
        }
    }
});

// Hook InternetConnectW to monitor network connections
var InternetConnectW = Module.findExportByName("wininet.dll", "InternetConnectW");
if (InternetConnectW) {
    Interceptor.attach(InternetConnectW, {
        onEnter: function(args) {
            console.log("[InternetConnectW] " + args[1].readUtf16String() +
                        ":" + args[2].toInt32());
        }
    });
}

Combining Tracing Tools

Recommended tracing strategy:

1. Start with strace -f -o trace.log ./sample
   → Get a high-level view of all system calls

2. Identify interesting behaviors (network, files, processes)

3. Use ltrace for library-level detail on specific calls
   → ltrace -e specific_function -s 256 ./sample

4. Use Frida for targeted, custom hooks on specific functions
   → Hook crypto functions to capture keys
   → Hook network functions to capture plaintext before encryption

References

Tools

Further Reading