In this paper, we propose a low-invasive framework to predict changes in cognitive performance using only heart rate variability (HRV). Although a lot of studies have tried to estimate cognitive performance using multiple vital data or electroencephalogram data, these methods are invasive for users because they force users to attach a lot of sensor units or electrodes to their bodies. To address this problem, we proposed a method to estimate cognitive performance using only HRV, which can be measured with as few as two electrodes. However, this can't prevent loss of worker productivity because the workers' productivity had already decreased even if their current cognitive performance had been estimated as being at a low level. In this paper, we propose a framework to predict changes in cognitive performance in the near future. We obtained three principal contributions in this paper: (1) An experiment with 45 healthy male participants clarified that changes in cognitive performance caused by mental workload can be predicted using only HRV. (2) The proposed framework, which includes a support vector machine and principal component analysis, predicts changes in cognitive performance caused by mental workload with 84.4 % accuracy. (3) Significant differences were found in some HRV features for test participants, depending on whether or not their cognitive performance changes had been predicted accurately. These results lead us to conclude that the framework has the potential to help both workers and managerial personnel predict what their performances will be in the near future. This will make it possible to proactively suggest rest periods or changes in work duties to prevent losses in productivity caused by decreases of cognitive work performance.