In this letter, we propose a simple framework for accelerating a state-of-the-art histogram-based weighted median filter at no expense. It is based on a process of determining the filter processing direction. The determination is achieved by measuring the local feature variation of input images. Through experiments with natural images, it is verified that, depending on input images, the filtering speed can be substantially increased by changing the filtering direction.