In this paper, we propose a robust statistical framework for extracting scenes from a baseball broadcast video. We apply multi-stream hidden Markov models (HMMs) to control the weights among different features. To achieve a large robustness against new scenes, we used a common simple structure for all the HMMs. In addition, scene segmentation and unsupervised adaptation were applied to achieve greater robustness against differences in environmental conditions among games. The F-measure of scene-extracting experiments for eight types of scene from 4.5 hours of digest data was 77.4% and was increased to 78.7% by applying scene segmentation. Furthermore, the unsupervised adaptation method improved precision by 2.7 points to 81.4%. These results confirm the effectiveness of our framework.