An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking

Ryoichi ISAWA; Daisuke INOUE; Koji NAKAO

doi:10.1587/transinf.2014EDP7268

An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking

Ryoichi ISAWA, Daisuke INOUE, Koji NAKAO

Full Text Views

0

Cite this

Summary :

Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.

Publication: IEICE TRANSACTIONS on Information Vol.E98-D No.4 pp.883-893

Publication Date: 2015/04/01

Publicized: 2015/01/06

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2014EDP7268

Type of Manuscript: PAPER

Category: Information Network

Authors

Ryoichi ISAWA
  National Institute of Information and Communications Technology (NICT)
Daisuke INOUE
  National Institute of Information and Communications Technology (NICT)
Koji NAKAO
  National Institute of Information and Communications Technology (NICT)

Keyword

software packer, malware, code analysis, Data Execution Prevention (DEP), security

Cite this

Copy

Ryoichi ISAWA, Daisuke INOUE, Koji NAKAO, "An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking" in IEICE TRANSACTIONS on Information, vol. E98-D, no. 4, pp. 883-893, April 2015, doi: 10.1587/transinf.2014EDP7268.
Abstract: Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2014EDP7268/_p

Copy

@ARTICLE{e98-d_4_883,
author={Ryoichi ISAWA, Daisuke INOUE, Koji NAKAO, },
journal={IEICE TRANSACTIONS on Information},
title={An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking},
year={2015},
volume={E98-D},
number={4},
pages={883-893},
abstract={Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.},
keywords={},
doi={10.1587/transinf.2014EDP7268},
ISSN={1745-1361},
month={April},}

Copy

TY - JOUR
TI - An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking
T2 - IEICE TRANSACTIONS on Information
SP - 883
EP - 893
AU - Ryoichi ISAWA
AU - Daisuke INOUE
AU - Koji NAKAO
PY - 2015
DO - 10.1587/transinf.2014EDP7268
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E98-D
IS - 4
JA - IEICE TRANSACTIONS on Information
Y1 - April 2015
AB - Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.
ER -