In this paper, we propose a simple nonlinear system which consists of a chaotic spiking oscillator and a controlling circuit to stabilize unknown periodic orbits. Our proposed system generates various stabilized unknown Unstable Periodic Orbits which are embedded on the chaotic attractor of the original chaotic spiking oscillator. The proposed system is simple and exhibits various bifurcation phenomena. The dynamics of the system is governed by 1-D piecewise linear return map. Therefore, the rigorous analysis can be performed. We provide conditions for stability and almost complete analysis for bifurcation and co-existence phenomena by using the 1-D return map. An implementation example of the controlled chaotic spiking oscillator is provided to confirm some theoretical results.