The spectrum handoff problem for cognitive radio systems is considered in this paper. The secondary users (SUs) can only opportunistically access the spectrum holes, i.e. the frequency channels unoccupied by the primary users (PUs). As long as a PU appears, SUs have to vacate the channel to avoid interference to PUs and switch to another available channel. In this paper, a prediction-based spectrum handoff scheme is proposed to reduce the negative effect (both the interference to PUs and the service block of SUs) during the switching time. In the proposed scheme, a hidden Markov model is used to predict the occupancy of a frequency channel. By estimating the state of the model in the next time instant, we can predict whether the frequency channel will be occupied by PUs or not. As a cross-layer design, the spectrum sensing performance parameters false alarm probability and missing detection probability are taken into account to enhance accuracy of the channel occupancy prediction. The proposed scheme will react on the spectrum sensing algorithm parameters while the spectrum handoff performance is significantly affected by them. The interference to the PUs could be reduced obviously by adapting the proposed spectrum handoff scheme, associated with a potential increase of switch delay of SUs. It will also be helpful for SUs to save broadband scan time and prefer an appropriate objective channel so as to avoid service block. Numerical results demonstrate the above performance improvement by using this prediction-based scheme.