This paper describes the theoretical analysis and experimental verification of a new type high-isolation surface-acoustic-wave (SAW) duplexer by using a SAW on-chip compensation circuit designed to cancel the signal of the main SAW filter at an attenuation frequency band. First, a numerical analysis based on the interference of waves propagating parallel waveguides is applied to clarify the relation between the absolute improvement value of the filter's attenuation level and cancel conditions. Then, the feasibility of the SAW compensation circuit using the double mode SAW (DMS) resonator filter are studied in both a circuit simulation and experiment. As a result, a 10-30 dB attenuation improvement was achieved within a band range of several tens of MHz using electrical characteristics of the lower side slope in the DMS resonator filter, and that it agrees well with the result obtained by numerical analysis. These results are expected to be useful for current and future mobile systems wants higher receiver sensitivity.