This paper presents a novel method to speed up neural network (NN) based face detection systems. NN-based face detection can be viewed as a classification and search problem. The proposed method formulates the face search problem as an integer nonlinear optimization problem (INLP) and expands the basic particle swarm optimization (PSO) to handle it. PSO works with a population of particles, each representing a subwindow in an input image. The subwindows are evaluated by how well they match a NN based face filter. A face is indicated when the filter response of the best particle is above a given threshold. Experiments on a set of 42 test images show the effectiveness of the proposed approach. Moreover, the effect of PSO parameter settings on the search performance was investigated.