We address the problem of measuring matching similarity in terms of template matching. A novel method called two-side agreement learning (TAL) is proposed which learns the implicit correlation between two sets of multi-dimensional data points. TAL learns from a matching exemplar to construct a symmetric tree-structured model. Two points from source set and target set agree to form a two-side agreement (TA) pair if each point falls into the same leaf cluster of the model. In the training stage, unsupervised weak hyper-planes of each node are learned at first. After then, tree selection based on a cost function yields final model. In the test stage, points are propagated down to leaf nodes and TA pairs are observed to quantify the similarity. Using TAL can reduce the ambiguity in defining similarity which is hard to be objectively defined and lead to more convergent results. Experiments show the effectiveness against the state-of-the-art methods qualitatively and quantitatively.