In this paper, the Viterbi decoder containing new branch metrics of the squared Euclidean distance with multiple order phase differences is introduced in order to improve the bit error rate (BER) in the differential detection of the trellis-coded MDPSK-OFDM. The proposed Viterbi decoder is conceptually same as the sliding multiple phase differential detection method that uses the branch metric with multiple phase differences. Also, we describe the Viterbi algorithm in order to use this branch metrics. Our study shows that such a Viterbi decoder improves BER performance without sacrificing bandwidth and power efficiency. Also, the proposed algorithm can be used in the single carrier modulation.