Autonomous driving technology is expected to be applied to various applications with unmanned vehicles (UVs), such as small delivery vehicles for office supplies and smart wheelchairs. UV remote control by a cloud server (CS) would achieve cost-effective applications with a large number of UVs. In general, dead time in real-time feedback control reduces the control accuracy. On remote path tracking control by the CS, UV control accuracy deteriorates due to transmission delay and jitter through the Internet. Digital twin computing (DTC) and jitter buffer are effective to solve this problem. In our previous study, we clarified effectiveness of them in UV remote control by CS. The jitter buffer absorbs the transmission delay jitter of control signals. This is effective to achieve accurate UV remote control. Adaptive buffering time optimization according to real-time transmission characteristics is necessary to achieve more accurate UV control in CS-based remote control system with DTC and jitter buffer. In this study, we proposed a method for the adaptive optimization according to real-time transmission delay characteristics. To quantitatively evaluate the effectiveness of the method, we created a UV remote control simulator of the control system. The results of simulations quantitatively clarify that the adaptive optimization by the proposed method improves the UV control accuracy.