Human detection has witnessed significant development in recent years. The introduction of cascade structure and integral histogram has greatly improved detection speed. But real-time detection is still only possible for sparse scan of 320 240 sized images. In this work, we propose a matrix-based structure to reorganize the computation structure of window-scanning detection algorithms, as well as a new pre-processing method called Hierarchical HOG Matrices (HHM) in place of integral histogram. Our speed-up scheme can process 320 240 sized images by dense scan (≈ 12000 windows per image) at the speed of about 30 fps, while maintaining accuracy comparable to the original HOG + cascade method.