A method is presented to perform image segmentation by accumulatively observing apparent motion in a long image sequence of a dynamic scene. In each image in the sequence, locations are grouped into small patches of approximately uniform optical flow. To reduce the noise in computed flow vectors, a local image motion vector of each patch is computed by averaging flow vectors in the corresponding patches in several images. A segment contains patches belonging to the same 3-D plane in the scene. Initial segments are obtained in the image, and then an attempt to merge or split segments is iterated to update the segments. In order to remove inherent ambiguities in motion-based segmentation, temporal coherence between the local image motion of a patch and the apprent motion of every plane is investigated over long time. In each image, a patch is grouped into the segment of the plane whose apparent motion is temporally most coherent with the local image motion of the patch. When apparent motions of two planes are temporally incoherent, segments of the planes are retained as individual ones.

A method for detecting multiple rigid motions in images from an optical flow field obtained with multi-scale, multi-orientation filters is proposed. Convolving consecutive gray scale images with a set of eight orientation-selective spatial Gaussian filters yields eight gradient constraint equations for the two components of a flow vector at every location. The flow vector and an uncertainty measure are obtained from these equations. In the neighborhood of motion boundary, the uncertainty of the flow vectors increase. By using multiple sets of filters of different scales, multiple flow vectors are obtained at every location, from which the one with minimal uncertainty measure is selected. The obtained flow field is then segmented in order to solve the aperture problem and to remove noise without blurring discontinuity in the flow field. Discontinuities are first detected as those locations where flow vectors have relatively larger uncertainty measures. Then similar flow vectors are gouped into regions. By modeling flow vectors, regions are merged to form segments each of which belongs to a planar patch of a rigid object in the scene.