We propose an accurate and scalable solution to the perspective-n-point problem, referred to as ASPnP. Our main idea is to estimate the orientation and position parameters by directly minimizing a properly defined algebraic error. By using a novel quaternion representation of the rotation, our solution is immune to any parametrization degeneracy. To obtain the global optimum, we use the Grobner basis technique to solve the polynomial system derived from the first-order optimality condition. The main advantages of our proposed solution lie in accuracy and scalability. Extensive experiment results, with both synthetic and real data, demonstrate that our proposed solution has better accuracy than the state-of-the-art noniterative solutions. More importantly, by exploiting vectorization operations, the computational cost of our ASPnP solution is almost constant, independent of the number of point correspondences n in the wide range from 4 to 1000. In our experiment settings, the ASPnP solution takes about 4 milliseconds, thus best suited for real-time applications with a drastically varying number of 3D-to-2D point correspondences.