Malware Analysis Fundamentals
Analyze malicious software safely — static analysis with strings, PE header inspection, and YARA rules; dynamic analysis with sandbox environments, Process Monitor, and network capture.
Learning Objectives
- → Perform static analysis: strings, file type, PE header inspection, entropy
- → Analyze malware behavior dynamically in an isolated sandbox
- → Use Process Monitor and Wireshark to observe malware actions
- → Write YARA rules to detect malware families
- → Classify malware: ransomware, RAT, rootkit, botnet, dropper
Malware Classification
Types of malware:
Virus — self-replicates by infecting other files
Worm — self-propagates across networks without host file
Trojan — masquerades as legitimate software
Ransomware — encrypts files, demands payment
RAT — Remote Access Trojan, full attacker control
Rootkit — hides itself and other malware from OS/AV
Botnet — C2-controlled network of compromised hosts
Dropper — downloads and installs other malware
Spyware — exfiltrates keystrokes, screenshots, credentials
Adware — displays unwanted advertisements
Fileless — runs only in memory, no disk artifact
Stages of malware execution:
1. Delivery (email, web, USB)
2. Exploitation (vulnerability or social engineering)
3. Installation (persistence mechanism)
4. C2 Communication (callbacks, beaconing)
5. Actions on Objective (data theft, encryption, lateral movement)
Lab Safety: Isolated Environment
NEVER analyze malware on your main machine!
Safe analysis environment:
1. Dedicated VM (VMware/VirtualBox) — snapshot before analysis
2. Host-only networking (no internet) for most analysis
3. Flare VM (Windows malware analysis VM):
https://github.com/mandiant/flare-vm
4. REMnux (Linux malware analysis):
https://remnux.org/
Before analysis:
- Take clean snapshot
- Disable shared folders (malware may escape VM)
- Block network (or use INetSim for fake internet)
- Document everything before touching the sample
After analysis:
- Revert to clean snapshot
- NEVER keep malware on a production machine
Static Analysis
File Type Identification
# Never trust file extension!
file suspicious.exe
# output: PE32 executable (GUI) Intel 80386, for MS Windows
file malware.pdf
# output: actually: ELF 64-bit LSB executable (if renamed)
# Calculate hash (fingerprint):
md5sum malware.exe
sha256sum malware.exe
# Search hash on VirusTotal: https://www.virustotal.com
# Entropy analysis (high entropy = packed/encrypted):
# Normal binary: entropy 5-7
# Packed/encrypted: entropy 7.5-8.0
python3 -c "
import math, sys
data = open(sys.argv[1], 'rb').read()
freqs = {}
for b in data:
freqs[b] = freqs.get(b, 0) + 1
entropy = -sum((c/len(data))*math.log2(c/len(data)) for c in freqs.values())
print(f'Entropy: {entropy:.2f}')
" malware.exe
Strings Extraction
# Extract printable strings from binary:
strings malware.exe
strings -n 10 malware.exe # min 10 chars
strings -e l malware.exe # 16-bit little-endian strings
# Look for:
# - URLs/IPs (C2 indicators)
# - File paths (where it drops files)
# - Registry keys (persistence)
# - Error messages (reveals functionality)
# - Encryption keys (hardcoded)
# - API function names (capability indication)
# - Domain names
# Filter interesting strings:
strings malware.exe | grep -E "(http|ftp|//|\.com|\.ru|\.cn)"
strings malware.exe | grep -iE "(password|decrypt|ransom|bitcoin|wallet)"
strings malware.exe | grep -E "HKEY" # registry keys
PE Header Analysis
# PE (Portable Executable) format — Windows executables
# Tools:
# - PE-bear (GUI)
# - pestudio (comprehensive static analysis)
# - pefile (Python library)
# - CFF Explorer
# Key sections to examine:
# .text — code (should be executable)
# .data — initialized data (should NOT be executable)
# .rsrc — resources (icons, strings, embedded files)
# .rdata — read-only data (strings, imports)
# Extra sections with high entropy = packed/encrypted payload
python3 << 'EOF'
import pefile
pe = pefile.PE('malware.exe')
# Imports (what Windows APIs it calls)
print("IMPORTS:")
for entry in pe.DIRECTORY_ENTRY_IMPORT:
dll = entry.dll.decode()
print(f" {dll}:")
for imp in entry.imports:
print(f" {imp.name.decode() if imp.name else 'ordinal'}")
# Sections
print("
SECTIONS:")
for section in pe.sections:
print(f" {section.Name.decode().strip()}: "
f"VirtualSize={section.Misc_VirtualSize}, "
f"Characteristics={hex(section.Characteristics)}")
EOF
# Suspicious imports indicate malware capabilities:
# CreateRemoteThread + VirtualAllocEx = process injection
# CryptEncrypt + FindFirstFile = ransomware
# RegSetValueEx + HKEY_CURRENT_USER = persistence
# InternetOpenUrl, socket = network activity
Dynamic Analysis
Sandbox Analysis
Online sandboxes (safe — no local infection risk):
- Any.run: https://any.run/ (interactive)
- Hybrid Analysis: https://www.hybrid-analysis.com/
- Joe Sandbox: https://www.joesandbox.com/
- VirusTotal: https://www.virustotal.com/gui/
Sandbox reports show:
- Process tree (what processes were created)
- File system changes (dropped files)
- Registry modifications (persistence)
- Network connections (C2 domains/IPs)
- Screenshots of execution
- Behavioral detection results
Process Monitor (Procmon)
Procmon (Windows Sysinternals) — monitors:
- File system operations (reads, writes, creates)
- Registry operations (sets, queries, creates)
- Network activity
- Process/thread events
Filter configuration for malware analysis:
1. Add filter: Process Name = malware.exe
2. Add filter: Operation = RegSetValue (persistence)
3. Add filter: Path contains APPDATA (common dropper location)
4. Add filter: Operation = WriteFile
What to look for:
- Dropped files in %TEMP%, %APPDATA%, %SystemRoot%
- Registry autorun keys:
HKCU\Software\Microsoft\Windows\CurrentVersion\Run
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
- Connection attempts to suspicious IPs
Network Traffic Analysis
# Use Wireshark during dynamic analysis:
# Or use INetSim (fake DNS/HTTP server) on the host-only network
# Common C2 indicators:
# - DNS queries for random-looking domains (DGA)
# - Regular beaconing (connection every 60s = C2 check-in)
# - HTTP POST with unusual user-agent
# - Large encrypted data exfiltration
# DNS exfiltration: data.base64data.attacker.com (encoded in subdomain)
# HTTP C2: User-Agent: Mozilla/5.0 (but with unique string)
# Wireshark filters:
dns # all DNS queries
http.request.method == "POST" # POST requests (data exfil)
tcp.flags.syn == 1 and tcp.flags.ack == 0 # new connections
YARA Rules
YARA: pattern-matching rules for malware detection
Used by: AV vendors, SIEM, sandbox, DFIR teams
```yara
rule Ransomware_Indicator {
meta:
description = "Detects common ransomware strings"
author = "Security Team"
date = "2024-01-01"
strings:
$ransom1 = "YOUR FILES HAVE BEEN ENCRYPTED" nocase
$ransom2 = "bitcoin" nocase
$ransom3 = ".locked" nocase
$ransom4 = "decrypt" nocase
$crypto = { 48 8B 05 ?? ?? ?? ?? } // byte pattern
condition:
2 of ($ransom*) or $crypto
}
Run YARA:
yara rule.yar malware.exe
yara -r rule.yar /suspicious/directory/ # recursive
yara -r /path/to/all/rules/ /tmp/malware/
YARA rule repositories:
https://github.com/Yara-Rules/rules
https://github.com/Neo23x0/signature-base
## Indicators of Compromise (IOCs)
IOCs: forensic artifacts indicating system compromise
Types of IOCs:
- File hashes (MD5, SHA256)
- Filenames and paths
- IP addresses
- Domain names
- URLs
- Registry keys
- Mutex names
- Network user-agents
IOC sharing formats:
STIX 2.0 — Structured Threat Information Expression
TAXII — Trusted Automated eXchange of Intelligence Information
MISP — Malware Information Sharing Platform
OpenIOC — Mandiant's IOC format
Extract IOCs with tools:
ioc-finder: extract IPs, domains, hashes from text
CAPE Sandbox: automated IOC extraction from behavior
```
Analyze a malware sample safely (use samples from MalwareBazaar: bazaar.abuse.ch — legal repository of malware samples for research): (1) NEVER run it — static only, (2) identify file type with 'file', (3) calculate SHA256 and search on VirusTotal, (4) run 'strings' and identify suspicious: IPs, domains, registry keys, filenames, (5) analyze entropy — is it packed? (6) use pefile to list imports — what capabilities do they suggest?
Why do malware authors pack their executables?
What does a call to CreateRemoteThread combined with VirtualAllocEx indicate?
Submit a malware sample to Any.run (any.run — interactive online sandbox, free tier available) or Hybrid Analysis: (1) upload a known malware sample (from MalwareBazaar — already public), (2) observe the process tree — what child processes were spawned, (3) identify all files created, (4) note all registry keys modified (especially Run keys), (5) record all network connections: IPs, domains, HTTP requests, (6) compile a mini-IOC report: hash, dropped files, C2 domains/IPs.
What are autorun registry keys and why does malware use them?
Write YARA rules to detect a malware family: (1) analyze 3 samples of the same malware family (use VirusTotal for strings), (2) identify common strings across all 3: ransom note text, C2 domain, file path, mutex name, (3) write a YARA rule with meta, strings, and condition sections, (4) test: yara rule.yar sample1.exe (should match), (5) test against clean file (should NOT match), (6) add a byte pattern for the malware's unique code sequence.
What makes a good YARA rule condition?