A modified frequency domain method for analyzing nonlinear waveform distortion in a magnetic recording process is presented. The measurement technique combines a 5th harmonic measurement technique, which uses a specific 30-bit pattern including dibits, and a precompensation technique for the dibits. The 5th harmonic voltage ratio given by the former technique includes the amount of NLTS (Nonlinear transition shift) and PE (Partial erasure) in dibits. The latter precompensation technique is employed to evaluate the PE as the minimum in the 5th harmonic voltage ratio. The true NLTS can be estimated from the amount of distortion and the evaluated PE. The high accuracy of the technique was confirmed by an examination using a pulse pattern generator with varied phase and amplitude. Finally, the effects of medium properties such as coercivity and squareness on the nonlinear distortions have been investigated by applying the technique to particulate flexible media. The NLTS increased with squareness from 3.5% to 7% while PE was less than 6% for any squareness at a recording density of 76 kFRPI. When coercivity became large, NLTS and PE decreased. The direction of NLTS for Ba-ferrite media agreed with that for a perpendicular Co-Cr thin-film medium.