Current software behavior models lack the ability to conduct semantic analysis. We propose a new model to detect abnormal behaviors based on a function semantic tree. First, a software behavior model in terms of state graph and software function is developed. Next, anomaly detection based on the model is conducted in two main steps: calculating deviation density of suspicious behaviors by comparison with state graph and detecting function sequence by function semantic rules. Deviation density can well detect control flow attacks by a deviation factor and a period division. In addition, with the help of semantic analysis, function semantic rules can accurately detect application layer attacks that fail in traditional approaches. Finally, a case study of RSS software illustrates how our approach works. Case study and a contrast experiment have shown that our model has strong expressivity and detection ability, which outperforms traditional behavior models.

In this paper, we propose a new trusted modeling approach based on state graphs. We introduce a novel method of deriving state-layer from a system call sequence in terms of probability and statistics theory, and we identify the state sequence with the help of Hidden Markov Model (HMM). We generate state transition graph according to software executing process and pruning rules. Then, we separate local function graphs according to software specific functions by semantic analysis. The state-layer is a bridge between the basic behaviors and the upper layer functions of software to compensate semantic faults. In addition, a pruning strategy of formulating state graphs is designed to precisely describe each piece of software functions. Finally, a detecting system based on our model is proposed, and a case study of RSS software reveals how our system works. The results demonstrate that our trusted model describes software behaviors successfully and can well detect un-trust behaviors, anomaly behaviors, and illegal input behaviors.