In this paper, a motion field representation algorithm based on directional edge information has been developed. This work is aiming at building an ego-motion detection system using dedicated VLSI chips developed for real time motion field generation at low powers . Directional edge maps are utilized instead of original gray-scale images to represent local features of an image and to detect the local motion component in a moving image sequence. Motion detection by edge histogram matching has drastically reduced the computational cost of block matching, while achieving a robust performance of the ego-motion detection system under dynamic illumination variation. Two kinds of feature vectors, the global motion vector and the component distribution vectors, are generated from a motion field at two different scales and perspectives. They are jointly utilized in the hierarchical classification scheme employing multiple-clue matching. As a result, the problems of motion ambiguity as well as motion field distortion caused by camera shaking during video capture have been resolved. The performance of the ego-motion detection system was evaluated under various circumstances, and the effectiveness of this work has been verified.