Current software behavior models lack the ability to conduct semantic analysis. We propose a new model to detect abnormal behaviors based on a function semantic tree. First, a software behavior model in terms of state graph and software function is developed. Next, anomaly detection based on the model is conducted in two main steps: calculating deviation density of suspicious behaviors by comparison with state graph and detecting function sequence by function semantic rules. Deviation density can well detect control flow attacks by a deviation factor and a period division. In addition, with the help of semantic analysis, function semantic rules can accurately detect application layer attacks that fail in traditional approaches. Finally, a case study of RSS software illustrates how our approach works. Case study and a contrast experiment have shown that our model has strong expressivity and detection ability, which outperforms traditional behavior models.