This release of OpenHIPS includes YARA support for scanning for metasploit shellcode. However, this doesn't do a good job of protecting against metasploit, as I'll explain.
In this release, I stripped out most of the HeapLocker functionality that was in here previously. I left in the check to see how much memory is being used which can potentially detect heap sprays (it suspends all threads in the process if the process's memory usage exceeds 400MB). I also left in the code that scans memory pages, but instead of using a single KMP search for the string "unpack" like HeapLocker does, I'm using YARA, and I'm scanning for 3 shell code patterns that are used by metasploit (the payloads for
shell_reverse_tcp). Unfortunately, I can't get these patterns to hit, so this release of OpenHIPS isn't going to secure you.
The problem with writing a HIPS like this, is you don't know what to scan and when. HeapLocker and currently OpenHIPS work by iterating through all of the memory allocations in a process repeatedly. In the first run through when the projects first get execution, they mark all the memory as "scanned" without actually scanning it, because if they scanned it, then they would get false positives because the memory containing the signatures to look for would get scanned, which would alert on the signatures. Also, you assume that the initial app doesn't have "shellcode" or whatever in it, so you don't scan it for efficiency and again to reduce the possibility of false positives. However, what this means is that if I run one of the msfpayload executables that just executes the payload without exploiting anything, then OpenHIPS won't detect the shellcode, even though it's in the binary plain as day, but it's not in new memory allocations which is where OpenHIPS looks.
Finally, I will run into problems if the shellcode is encoded or obfuscated. Yuck.
Anyway, expect a new release in a week or two that hopefully does a better job of catching metasploit.
Source code | Installer | Debug Installer