An optimum representative-frames (r-frames) selection method using step-wise function approximation has been developed to provide automatic indexing for a video-query-agent (VQA) system. It uses dynamic programming to simultaneously select the r-frames and corresponding segment boundaries. Experiments showed that the approximation error of the selected r-frames was less than that of two conventional methods. Retrieval experiments using a feature-based image-search engine showed that the proposed method is more robust and effective than the two conventional methods. The proposed method was implemented in a VQA system and processing time was evaluated. The results showed that the processing time for indexing was shorter than that of the conventional method.

In this paper, visualized sound retrieval and categorization methods using a feature-based image search engine were evaluated aiming at efficient video scene query. Color-coded patterns of the sound spectrogram are adopted as the visualized sound index. Sound categorization experiments were conducted using visualized sound databases including speech, bird song, musical sounds, insect chirping, and the sound-track of sports video. The results of the retrieval experiments show that the simple feature-based image search engine can be effectively used for visualized sound retrieval and categorization. The results of categorization experiments involving humans show that after brief training humans can at least do rough categorization. These results suggest that using visualized sound can be effective method for an efficient video scene query.