Adaptive maximum likelihood differential detection implemented by a reduced state-transition Viterbi algorithm (called adaptive 3-state RSTVDD) is presented for adaptive reception of M-ary differential phase shift keying (DPSK) signals transmitted over additive white Gaussian noise (AWGN) and frequency-nonselective Rayleigh fading channels. The adaptive 3-state RSTVDD comprises 1DD, a differential encoder, and reverse modulator, followed by reduced-state (3-state) Viterbi DD (RSVDD) with adaptive phase reference estimation. The adaptive 3-state RSVDD detector estimates the sequence of phase errors of the 1DD output. The phase reference estimator is an adaptive least mean square (LMS) filter with a step-size that adapts to changing channel conditions. The final detected symbol sequence is the modulo-2π sum of the 1DD output phase sequence and the detected phase error sequence. The bit error rate (BER) performance of M-ary DPSK, M=4, 8, and 16, in the AWGN and Rayleigh fading channels is evaluated by computer simulation to show that adaptive 3-state RSTVDD can achieve almost the same BER performance as the previously developed adaptive M-state RSVDD. Since the number of trellis states is reduced to three irrespective of M, the adaptive 3-state RSTVDD has lower computation complexity and it is particularly useful for M-ary DPSK with M8.