Autonomous driving is not only required to detect pedestrians around vehicles, but also expected to understand the behaviors of pedestrians. Pedestrian body orientation and head orientation are the relevant indicators of the pedestrian intention. This paper proposes an accurate estimation system to recognize the pedestrian body orientation and the pedestrian head orientation from on-board camera and inertial sensors. The proposed system discretizes the body orientation and the head orientation into 16 directions. In order to achieve the accurate orientation estimation, a novel training database is established, which includes strongly labeled data and weakly labeled data. Semi-Supervised Learning method is employed to annotate the weakly labeled data, and to generate the accurate classifier based on the proposed training database. In addition, the temporal constraint and the human physical model constraint are considered in orientation estimation, which are beneficial to the reasonable and stable result of orientation estimation for the pedestrian in image sequences. This estimated result is the orientation in camera space. The comprehension of the pedestrian behavior needs to be conducted in the real world space. Therefore, this paper proposes to model the motion of the host vehicle using inertial sensor, then transforms the estimated orientation from camera space to the real world space by considering the vehicle and pedestrian motion. The represented orientation indicates the behavior of the pedestrian more directly. Finally, a series of experiments demonstrate the effectiveness of the proposed pedestrian orientation system.