This paper proposes an efficient two-phase optimization approach for a compact W-band double-plane stepped rectangular waveguide filter design, which combines genetic algorithms (GAs) with the simplified transmission-line model and full-wave analysis. Being more efficient and robust than the gradient-based method, the approach can lead to a compact waveguide filter design. Numerical results show that the resultant waveguide filter design with 4 sections (total length 19.6 mm) is sufficient to meet the design goal and provides comparable performance to that with 8 sections (total length 35.6 mm) by the Chebyshev synthesis approach. Based on the present approach, nineteen compact high-pass waveguide filters have been implemented and measured at the W-band with satisfactory performance.