Although the classical vector median filter (VMF) has been widely used to suppress the impulse noise in the color image, many thin color curve pixels aligned in arbitrary directions are usually removed out as impulse noise. This serious problem can be solved by the proposed method that can protect the thin curves in arbitrary direction in color image and remove out the impulse noise at the same time. Firstly, samples in the 3x3 filter window are considered to preliminarily detect whether the center pixel is corrupted by impulse noise or not. Then, samples outside a 5x5 filter window are conditionally and partly considered to accurately distinguish the impulse noise and the noise-free pixel. At last, based on the previous outputs, samples on the processed positions in a 3x3 filter window are chosen as the samples of VMF operation to suppress the impulse noise. Extensive experimental results indicate that the proposed algorithm can be used to remove the impulse noise of color image while protecting the thin curves in arbitrary directions.