Static/Signature Detection
A signature is a set of unique bytes or strings that identify a piece of malware; other conditions can be variables or function names. Once the security solution scans the program, it compares it against a list of rules (YARA). For example, if a shellcode starts with FC 48 83 E4 F0 E8 C0 00 00 00 41 51 41 50 52 51 we know it is an msfvenom payload
Hash Detection
Part of static/signature detection, and the easiest and fastest way to detect malware: simply storing the hashes (MD5, SHA256…) of known malware in a database
Heuristic Detection
Since signature detection can easily be bypassed by changing a few bytes or tweaking the code, this was created:
Static Heuristic Detection
It decompiles the program and compares chunks of code with known malware stored in the AV's own databases
Dynamic Heuristic Analysis (SandBox)
It analyses malware behaviour by uploading it to its cloud server and running it; once there, it analyses its behaviour and flags it as good or not
(This can be anticipated by making it run only benign code)
Behavior-Based Detection
Once the malware is running, security solutions look for suspicious behaviour. For example, if it injects code into the notepad.exe process and connects to the Internet, that will make it suspicious
The best way to get past it is to avoid detectable actions as much as possible (spawning a cmd.exe, etc.)
Api Hooking
Mostly used by EDRs: they intercept the most used APIs and analyse the parameters passed to them in real time
IAT Checking
We can see which DLLs and functions a PE uses. For example, ransomware uses CreateFileA/W, SetFilePointer, Read/WriteFile, CryptCreateHash, CryptHashData and CryptGetHashParam, and that is detectable
Manual Analysis
The blue team can analyse the malware manually through reverse engineering, although anti-reversing techniques can be implemented