Facial skin detection is an important step in facial surgical planning like as many other applications. There are many problems in facial skin detection. One of them is that the image features can be severely corrupted due to illumination, noise, and occlusion, where, shadows can cause numerous strong edges. Hence, in this paper, we present an automatic Expectation-Maximization (EM) algorithm for facial skin color segmentation that uses knowledge of chromatic space and varying illumination conditions to correct and segment frontal and lateral facial color images, simultaneously. The proposed EM algorithm leads to a method that allows for more robust and accurate segmentation of facial images. The initialization of the model parameters is very important in convergence of algorithm. For this purpose, we use a method for robust parameter estimation of Gaussian mixture components. Also, we use an additional class, which includes all pixels not modeled explicitly by Gaussian with small variance, by a uniform probability density, and amending the EM algorithm appropriately, in order to obtain significantly better results. Experimental results on facial color images show that accurate estimates of the Gaussian mixture parameters are computed. Also, other results on images presenting a wide range of variations in lighting conditions, demonstrate the efficiency of the proposed color skin segmentation algorithm compared to conventional EM algorithm.