Spectrum and Bit Error Rate (BER) performance of a coded π/2 phase shift Non-Square (NS) 8 Quadrature Amplitude Modulation (QAM) system are studied in this letter. The modulation process of this scheme removes all 180 phase shift between adjacent constellation points and contains inherent memory which can be treated as a type of inner coding in coded system. Simulation results show that this modulation scheme has much lower spectrum regrowth and better BER performance when passing through nonlinear channel compared with conventional mode.