A deterministic simulation method is investigated for rate 3/4 trellis-coded (TC) 16PSK, TC-16QAM and TC partially overlapped (PO) 12QAM schemes. The BER performance of these schemes are evaluated on realistic channels with the same transmisson parameters, and are compared with BER values obtained from a distance spectrum expression. The results of this study show that the BER values obtained by the deterministic simulation coincide well with those from the distance spectrum method, and that TC-16PSK is superior to both TC-16QAM and TC-PO-12QAM in terms of BER performance over linear and nonlinear channels. The BER degradation caused by a nonlinear amplifier and the BER difference between two mapping methods ( Gray code and natural binary code mapping) are also clarified.