Mode-by-mode impulse responses, or spectral transfer matrix (STM) of birefringent fibers are measured by using linear optical sampling, with assist of polarization multiplexed probe pulse. By using the eigenvalue analysis of the STM, the differential mode delay and PMD vector of polarization-maintaining fiber are analyzed as a function of optical frequency over 1THz. We show that the amplitude averaging of the complex impulse responses is effective for enhancing the signal-to-noise ratio of the measurement, resulting in improving the accuracy and expanding the bandwidth of the measurement.