The Hough voting framework is a popular approach to parts based pedestrian detection. It works by allowing image features to vote for the positions and scales of pedestrians within a test image. Each vote is cast independently from other votes, which allows for strong occlusion robustness. However this approach can produce false pedestrian detections by accumulating votes inconsistent with each other, especially in cluttered scenes such as typical street scenes. This work aims to reduce the sensibility to clutter in the Hough voting framework. Our idea is to use object segmentation and object pose parameters to enforce votes' consistency both at training and testing time. Specifically, we use segmentation and pose parameters to guide the learning of a pedestrian model able to cast mutually consistent votes. At test time, each candidate detection's support votes are looked upon from a segmentation and pose viewpoints to measure their level of agreement. We show that this measure provides an efficient way to discriminate between true and false detections. We tested our method on four challenging pedestrian datasets. Our method shows clear improvements over the original Hough based detectors and performs on par with recent enhanced Hough based detectors. In addition, our method can perform segmentation and pose estimation as byproducts of the detection process.