Community detection is an important task in the social network analysis field. Many detection methods have been developed; however, they provide little semantic interpretation for the discovered communities. We develop a framework based on joint matrix factorization to integrate network topology and node content information, such that the communities and their semantic labels are derived simultaneously. Moreover, to improve the detection accuracy, we attempt to make the community relationships derived from two types of information consistent. Experimental results on real-world networks show the superior performance of the proposed method and demonstrate its ability to semantically annotate communities.