As more and more software vulnerabilities are exposed, shellcode has become very popular in recent years. It is widely used by attackers to exploit vulnerabilities and then hijack program's execution. Previous solutions suffer from limitations in that: 1) Some methods based on static analysis may fail to detect the shellcode using obfuscation techniques. 2) Other methods based on dynamic analysis could impose considerable performance overhead. In this paper, we propose Lemo, an efficient shellcode detection system. Our system is compatible with commodity hardware and operating systems, which enables deployment. To improve the performance of our system, we make use of the multi-core technology. The experiments show that our system can detect shellcode efficiently.

The shellcode use of the polymorphic form has become active as the de facto method for avoiding signature based network security system. We present a new static analysis method for detecting the decryption routine of the polymorphic shellcode. This method traces the processes by which the decryption routine stores the current program counter in a stack, moves the value between registers and uses the value in order to make the address of the encrypted code accessible. Most of decryption routines have the feature which they use the program counter stored on a stack as the address for accessing the memory that the encrypted code is positioned.