Parameter estimation theorems for LFM signals have been developed due to the advantages of fractional Fourier transform (FrFT). The traditional estimation methods in the fractional Fourier domain (FrFD) are almost based on two-dimensional search which have the contradiction between estimation performance and complexity. In order to solve this problem, we introduce the orthogonal matching pursuit (OMP) into the FrFD, propose a modified optimization method to estimate initial frequency and final frequency of fractional bandlimited LFM signals. In this algorithm, the differentiation fractional spectrum which is used to form observation matrix in OMP is derived from the spectrum analytical formulations of the LFM signal, and then, based on that the LFM signal has approximate rectangular spectrum in the FrFD and the correlation between the LFM signal and observation matrix yields a maximal value at the edge of the spectrum (see Sect.3.3 for details), the edge spectrum information can be extracted by OMP. Finally, the estimations of initial frequency and final frequency are obtained through multiplying the edge information by the sampling frequency resolution. The proposed method avoids reconstruction and the traditional peak-searching procedure, and the iterations are needed only twice. Thus, the computational complexity is much lower than that of the existing methods. Meanwhile, Since the vectors at the initial frequency and final frequency points both have larger modulus, so that the estimations are closer to the actual values, better normalized root mean squared error (NRMSE) performance can be achieved. Both theoretical analysis and simulation results demonstrate that the proposed algorithm bears a relatively low complexity and its estimation precision is higher than search-based and reconstruction-based algorithms.