VoIP, one of emerging technologies, offers high quality of real time voice services over IP-based broadband networks; however, the quality of voice would easily be degraded by IP network impairments such as delay, jitter and packet loss, hereon initiate the presence of new technologies to help solve out the problems. Among those, playout buffer at the receiving end can compensate for the jitter effects by its function of tradeoff between delay and loss. Adaptive smoothing algorithms are capable of the dynamical adjustment of smoothing size by introducing a variable delay based on the use of the network parameters so as to avoid the quality decay problem. This paper introduces an efficient and feasible perceived quality method for buffer optimization to achieve the best voice quality. This work formulates an online loss model which incorporates buffer sizes and applies the ITU-T E-model approach to optimize the delay-loss problem. Distinct from other optimal smoothers, the proposed optimal smoother can be applied for most codecs and carries the lowest complexity. Since the adaptive smoothing scheme introduces variable playback delays, the buffer re-synchronization between the capture and the playback becomes essential. This work also presents a buffer re-synchronization algorithm based on silence skipping to prevent unacceptable increase in the buffer preloading delay and even buffer overflow. Simulation experiments validate that the proposed adaptive smoother achieves significant improvement in the voice quality.