A Trellis-Coded 8PSK (TCM) modem has been developed. This modem can transmit HDTV programs at a bit rate of 60Mbps over a satellite channel. In addition to Trellis coding, the (255, 235) Reed-Solomon code is adopted together with an interleaving technique to improve bit error performance. As a result, it has been confirmed that bit error rate of 10-10 is achievable at a carrier to noise ratio of around 8.5dB over a 30MHz Gaussian noise channel. This TCM modem is also designed to double as a DQPSK modem. We examined the performances of these two modulation methods in a nonlinear channel. It was found that TCM is less disturbed than DQPSK by nonlinear distortion due to TWT. In addition, a limiter amplifier cascaded to TWT improves DQPSK performance and disturbs TCM performance. However it was confirmed that TCM still has a coding gain of 3dB over DQPSK.