In this paper, we report an experimental and numerical analysis of ultrahigh-speed coherent Nyquist pulse transmission. First, we describe a low-nonlinearity dispersion compensator for ultrahigh-speed coherent Nyquist pulse transmission; it is composed of a chirped fiber Bragg grating (CFBG) and a liquid crystal on silicon (LCoS) device. By adopting CFBG instead of inverse dispersion fiber, the nonlinearity in a 160km transmission line was more than halved. Furthermore, by eliminating the group delay fluctuation of the CFBG with an LCoS device, the residual group delay was reduced to as low as 1.42ps over an 11nm bandwidth. Then, by using the transmission line with the newly constructed low-nonlinearity dispersion compensator, we succeeded in improving the BER performance of single-channel 15.3Tbit/s-160km transmission by one-third compared with that of a conventional dispersion-managed transmission line and obtained a spectral efficiency of 8.7bit/s/Hz. Furthermore, we numerically analyzed the BER performance of its Nyquist pulse transmission. The numerical results showed that the nonlinear impairment in the transmission line is the main factor limiting the transmission performance in a coherent Nyquist pulse transmission, which becomes more significant at higher baud rates.