Heterogeneous networks (HetNets) are emerging as an inevitable method to tackle the capacity crunch of the cellular networks. Due to the complicated network environment and a large number of configured parameters, coverage and capacity optimization (CCO) is a challenging issue in heterogeneous cellular networks. By combining the self-optimizing algorithm for radio frequency (RF) parameters with the power control mechanism of small cells, the CCO problem of self-organizing network is addressed in this paper. First, the optimization of RF parameters is solved based on reinforcement learning (RL), where the base station is modeled as an agent that can learn effective strategies to control the tunable parameters by interacting with the surrounding environment. Second, the small cell can autonomously change the state of wireless transmission by comparing its distance from the user equipment with the virtual cell size. Simulation results show that the proposed algorithm can achieve better performance on user throughput compared to different conventional methods.