Railway inspection is important in railway maintenance. There are several tasks in railway inspection, e.g., defect detection and bolt detection. For those inspection tasks, the detection of rail surface is a fundamental and key issue. In order to detect rail defects and missing bolts, one must know the exact location of the rail surface. To deal with this problem, we propose an efficient Rail Surface Detection (RSD) algorithm that combines boundary and region information in a uniform formulation. Moreover, we reevaluate the rail location by introducing the top down information–bolt location prior. The experimental results show that the proposed algorithm can detect the rail surface efficiently.