A novel nonlinear analysis method of high power amplifier instability has been developed. This analysis method deals with a loop oscillation in a closed loop circuit and presents the conditions for oscillation under large-signal operation by taking account of mixing effect of FETs. Applying this analysis to the high power amplifier instability that an output power for the fundamental wave (f0-wave) decreases at some compression point where a half of the fundamental wave (f0/2-wave) is observed, it has been found that this instability is caused by an f0/2 loop oscillation. In addition, it has been verified by analysis and experiment that the oscillation can be removed by employing an isolation resistor in a closed loop circuit.