In this paper, a new Hierarchical Sum of Double Difference metric, HSDD, is introduced. It is shown, as opposed to conventional Sum of Absolute Difference (SAD) metric, how this zerotree coding aware metric can jointly constrain the motion vector searching for both temporal and spatial (quad-tree) directions under multiresolution motion estimation framework. The reward from the temporal-spatial co-optimization concept of HSDD is that fewer bits are spent later for describing the isolated zeros. The embedded wavelet video coder using HSDD metric was tested with a set of video sequences and the compression performance seems to be promising.