In this paper, an alternative transform based on the correlation of the residual block is proposed for the improvement of the H.264/AVC coding efficiency. A discrete sine transform is used alternately with a discrete cosine transform in order to greatly compact the energy of the signal when the correlation coefficients of the signal are in the range of -0.5 to 0.5. Therefore, the discrete sine transform is suggested to be used in conjunction with the discrete cosine transform in H.264/AVC. The alternative transform selecting the optimal transform between two transforms by using rate-distortion optimization shows a coding gain compared with H.264/AVC. The proposed method achieves a PSNR gain of up to 1.0 dB compared to JM 10.2 at relatively high bitrates.

In H.264/AVC, the quantized coefficients are scanned in a zigzag pattern. But the zigzag scanning is not always efficient for the directional spatial predictions in the intra coding of H.264/AVC. In this letter, we propose an adaptive scanning using the pixel similarity of the neighboring pixels to achieve enhanced intra coding performance. The proposed method reduces the bit rate approximately 2% compared with H.264/AVC without video quality degradation.

In order to reduce spatial redundancies, the H.264/AVC Intra coding provides nine directional prediction modes including DC prediction for every 44 block, but it needs a lot of overhead bits to represent the nine directional prediction modes for every 44 block. To compress the directional mode bits efficiently, the most probable mode is estimated by using the correlation between the prediction mode of spatially adjacent blocks and that of the current block. In this paper, a new method for estimating the most probable mode is proposed by using the directional information of the prediction mode of the adjacent blocks. Experimental results show that the proposed method is able to achieve a coding gain of about 0.2 dB on average at low bit rate.