Head-mounted displays (HMDs) and augmented reality (AR) are actively being studied. However, ordinary AR HMDs for visual assistance have a problem in which users have difficulty simultaneously focusing their eyes on both the real target object and the displayed image because the image can only be displayed at a fixed distance from an user's eyes in contrast to where the real object three-dimensionally exists. Therefore, we considered incorporating a holographic technology, an ideal three-dimensional (3D) display technology, into an AR HMD system. A few studies on holographic HMDs have had technical problems, and they have faults in size and weight. This paper proposes a compact holographic AR HMD system with the purpose of enabling an ideal 3D AR HMD system which can correctly reconstruct the image at any depth. In this paper, a Fourier transform optical system (FTOS) was implemented using only one lens in order to achieve a compact and lightweight structure, and a compact holographic AR HMD system was constructed. The experimental results showed that the proposed system can reconstruct sharp images at the correct depth for a wide depth range. This study enabled an ideal 3D AR HMD system that enables simultaneous viewing of both the real target object and the reconstructed image without feeling visual fatigue.