Transmission performance of amplitude and phase shift keying (APSK) format is studied theoretically. The extinction ratio of the amplitude shift keying (ASK) signal of the APSK format causes a trade-off of the performance between the ASK and the phase shift keying (PSK) signal of the APSK format. Then, zero-nulling method is proposed to improve the performance of the APSK format, and its effectiveness is confirmed by the numerical simulation.