The ability to recognize quickly inside network flows to be executable is prerequisite for malware detection. For this purpose, we introduce an instruction transition probability matrix (ITPX) which is comprised of the IA-32 instruction sets and reveals the characteristics of executable code's instruction transition patterns. And then, we propose a simple algorithm to detect executable code inside network flows using a reference ITPX which is learned from the known Windows Portable Executable files. We have tested the algorithm with more than thousands of executable and non-executable codes. The results show that it is very promising enough to use in real world.

The shellcode use of the polymorphic form has become active as the de facto method for avoiding signature based network security system. We present a new static analysis method for detecting the decryption routine of the polymorphic shellcode. This method traces the processes by which the decryption routine stores the current program counter in a stack, moves the value between registers and uses the value in order to make the address of the encrypted code accessible. Most of decryption routines have the feature which they use the program counter stored on a stack as the address for accessing the memory that the encrypted code is positioned.