In this letter, we propose an algorithm to estimate the optical flow fields based on a hierarchical structure composed of spatio-temporal image pyramids obtained from repetitive application of the Gaussian filtering and decimation in both the spatial and temporal domain. In our approach, an inter-level motion smoothness constraint between adjacent pyramid levels is introduced to estimate a unique optical flow field. We show that the pyramid structure allows us to employ the multigrid algorithm, which is known to accelerate the convergence rate. The multigrid algorithm provides a scheme for efficient combination of local and global information to estimate the optical flow field. The experimental results reveal that the combination of local and global information yields a fast convergence behavior and accurate motion estimation results.