Recently, a malicious user attacks a web browser through a malicious page that exploits the vulnerability of the browser and that executes malicious code. To prevent this attack, some methods have been devised such as DEP (Data Execution Prevention) that prevents data in stack frame or heap region from being executed. However, to evade these defense techniques, return-oriented programming (ROP) is introduced. ROP executes arbitrary code indirectly using gadget, which is group of instructions including ret instruction in a module that doesn't apply ASLR (Address Space Layout Randomization). In this paper, we propose a static approach to detect ROP payload in a network irrespective of the environment of the system under attack. Most studies have tried to detect ROP attacks using dynamic analysis, because ROP has various addresses of gadgets according to loaded modules. These methods have a limitation that must consider the environment of system to operate ROP, such as the version of OS and modules including gadgets. To overcome this limitation, our method detects ROP payload using static analysis without preliminary knowledge about the environment. We extract five characteristics of ROP and then propose a novel algorithm, STROP, to detect ROP in payload without execution. Our idea is as follows: STROP makes stack frame using input payload statically. It extracts addresses suspected as indicating gadgets and makes groups using the addresses. And then, STROP determine whether the payload includes ROP based on static characteristics. We implement a prototype using snort (network-based intrusion system) and evaluate it. Experiments show that our technique can detect ROP payload with a low number of false alarms. False positive (FP) is 1.3% for 2,239 benign files and 0.05-0.51% for 1GB packet dump file. Among 68 ROP payloads, STROP detects 51 payloads. This research can be applied to existing systems that collect malicious codes, such as Honeypot.

The growing use of web services is increasing web browser attacks exponentially. Most attacks use a technique called heap spraying because of its high success rate. Heap spraying executes a malicious code without indicating the exact address of the code by copying it into many heap objects. For this reason, the attack has a high potential to succeed if only the vulnerability is exploited. Thus, attackers have recently begun using this technique because it is easy to use JavaScript to allocate the heap memory area. This paper proposes a novel technique that detects heap spraying attacks by executing a heap object in a real environment, irrespective of the version and patch status of the web browser. This runtime execution is used to detect various forms of heap spraying attacks, such as encoding and polymorphism. Heap objects are executed after being filtered on the basis of patterns of heap spraying attacks in order to reduce the overhead of the runtime execution. Patterns of heap spraying attacks are based on analysis of how an web browser accesses benign web sites. The heap objects are executed forcibly by changing the instruction register into the address of them after being loaded into memory. Thus, we can execute the malicious code without having to consider the version and patch status of the browser. An object is considered to contain a malicious code if the execution reaches a call instruction and then the instruction accesses the API of system libraries, such as kernel32.dll and ws_32.dll. To change registers and monitor execution flow, we used a debugger engine. A prototype, named HERAD(HEap spRAying Detector), is implemented and evaluated. In experiments, HERAD detects various forms of exploit code that an emulation cannot detect, and some heap spraying attacks that NOZZLE cannot detect. Although it has an execution overhead, HERAD produces a low number of false alarms. The processing time of several minutes is negligible because our research focuses on detecting heap spraying. This research can be applied to existing systems that collect malicious codes, such as Honeypot.

The software birthmarking technique has conventionally been studied in fields such as software piracy, code theft, and copyright infringement. The most recent API-based software birthmarking method (Han et al., 2014) extracts API call sequences in entire code sections of a program. Additionally, it is generated as a birthmark using a cryptographic hash function (MD5). It was reported that different application types can be categorized in a program through pre-filtering based on DLL/API numbers/names. However, similarity cannot be measured owing to the cryptographic hash function, occurrence of false negatives, and it is difficult to functionally categorize applications using only DLL/API numbers/names. In this paper, we propose an API-based software birthmarking method using fuzzy hashing. For the native code of a program, our software birthmarking technique extracts API call sequences in the segmented procedures and then generates them using a fuzzy hash function. Unlike the conventional cryptographic hash function, the fuzzy hash is used for the similarity measurement of data. Our method using a fuzzy hash function achieved a high reduction ratio (about 41% on average) more than an original birthmark that is generated with only the API call sequences. In our experiments, when threshold ε is 0.35, the results show that our method is an effective birthmarking system to measure similarities of the software. Moreover, our correlation analysis with top 50 API call frequencies proves that it is difficult to functionally categorize applications using only DLL/API numbers/names. Compared to prior work, our method significantly improves the properties of resilience and credibility.

Return-oriented programming (ROP) attacks, which have been increasing in number recently, are an exploitation technique that can bypass non-executable page protection methods by using codes that exist within benign programs or modules. There have been many studies on defense against ROP attacks, but most of them have high overhead or high time complexity in terms of the detection of gadgets. In this letter, we suggest an ROP defense technique which is fast, space-efficient, and of lower detection time complexity; it uses a compiler-based approach. The most recent ROP defense technique is a compiler-based zero-sum defender suggested by Kim et al., achieving very low overhead. However, it still did not solve the issue of time complexity regarding detection. Our technique performs a specific computation to identify gadgets at the resetting position immediately before and after a return instruction. This method can efficiently identify a series of gadgets performed without calls and defend against them. In our experiment, the performance overhead was 1.62% and the file size overhead was 4.60%; our proposed technique achieved O(1) in terms of time complexity while having almost the same overhead as the zero-sum defender.

Jeong et al. recently have proposed a strong ID-based key distribution scheme in order to achieve security against long-term key reveal and session state reveal attacks. In this letter, we show that, unfortunately, the ID-based key distribution scheme is vulnerable to an impersonation attack such that anyone can manipulate public transcripts generated by a user to impersonate the original user.