Ground truth based image segmentation evaluation paradigm plays an important role in objective evaluation of segmentation algorithms. So far, many evaluation methods in terms of comparing clusterings in machine learning field have been developed. However, most traditional pairwise similarity measures, which only compare a machine generated clustering to a “true” clustering, have their limitations in some cases, e.g. when multiple ground truths are available for the same image. In this letter, we propose utilizing an information theoretic measure, named NJMI (Normalized Joint Mutual Information), to handle the situations which the pairwise measures can not deal with. We illustrate the effectiveness of NJMI for both unsupervised and supervised segmentation evaluation.