A novel amplification mechanism of traveling TM wave on an electron beam within a waveguide structure is proposed. Under boundary constraint of the waveguide, a hybrid coupling of longitudinal plasma wave and transverse guided one occurs to result in traveling instability. The instability refers to a backward traveling amplification. The new amplification in the waveguide due to the interactive coupling between the space charge mode and the waveguide one is firstly pointed out. The analysis is extended to the relativistic energy range to get a large gain. The features and properties are discussed for a wide frequency range as well as a high gain-bandwidth product.