Static Detection
Signatures and rules
For example, YARA rules are used by most analysts: if there is a rule that looks for VirtualAlloc, it will match
Hash Detection
The simplest approach: hash the malware and compare it against databases
PE Metadata Detection
For example, ransomware uses APIs such as CreateFileA/W, SetFilePointer, ReadFile/WriteFile and CryptCreateHash, which makes the analyst suspicious. It can be analysed with dumpbin.exe
Reputation-based Detection
Microsoft Defender SmartScreen evaluates downloads, URLs, apps and signing certificates, and even if nothing matches known malware it can still trigger because there is no established reputation
Cloud Analysis
EDRs and AVs can use the cloud to analyse the malware and return a final verdict on whether it is dangerous
Machine Learning Detection
Models analyse characteristics such as running processes and scripts to judge how suspicious they are
Heuristic Detection
Useful because malware can be changed, rebuilt, packed or obfuscated. The malware is uploaded to a sandbox that runs it and watches what it does
Memory Scanning
It looks for suspicious patterns, strings, executable private memory and RWX regions
API Monitoring and API Hooking
It watches how a process interacts with the operating system through the functions exposed by DLLs such as kernel32.dll, KernelBase.dll, advapi32.dll and ntdll.dll.
For example, WriteProcessMemory is not bad in itself, but the relationship between the source and target processes, and the memory protection, can give signals.
AMSI
ETW And Event Telemetry
Windows monitoring used by applications, services and the OS, which produce event telemetry.
It can be used to observe activity such as process creation, threads, image loads, file activity, registry activity, network activity, PowerShell, authentication and the kernel
Sysmon and Endpoint Logs
A Microsoft monitoring tool that records Windows system detail in the Windows Event Log, and can record activity like that in the previous point
Manual Analysis
Manual malware analysis
