We present an accurate and easy-to-use multi-sensor fusion toolbox for autonomous vehicles. It includes a ‘target-less’ multi-LiDAR (Light Detection and Ranging), and Camera-LiDAR calibration, sensor fusion, and a fast and accurate point cloud ground classifier. Our calibration methods do not require complex setup procedures, and once the sensors are calibrated, our framework eases the fusion of multiple point clouds, and cameras. In addition we present an original real-time ground-obstacle classifier, which runs on the CPU, and is designed to be used with any type and number of LiDARs. Evaluation results on the KITTI dataset confirm that our calibration method has comparable accuracy with other state-of-the-art contenders in the benchmark.

A multi-camera setup for a surveillance system enables a larger coverage area, especially when a single camera has limited monitoring capability due to certain obstacles. Therefore, for large-scale coverage, multiple cameras are the best option. In this paper, we present a method for detecting multiple objects using several cameras with large overlapping views as this allows synchronization of object identification from a number of views. The proposed method uses a graph structure that is robust enough to represent any detected moving objects by defining their vertices and edges to determine their relationships. By evaluating these object features, represented as a set of attributes in a graph, we can perform lightweight multiple object detection using several cameras, as well as performing object tracking within each camera's field of view and between two cameras. By evaluating each vertex hierarchically as a subgraph, we can further observe the features of the detected object and perform automatic separation of occluding objects. Experimental results show that the proposed method would improve the accuracy of object tracking by reducing the occurrences of incorrect identification compared to individual camera-based tracking.

Recently, cameras are equipped on cars in order to assist their drivers. These cameras often have a severe radial distortion because of their wide view angle, and sometimes it is necessary to compensate it in a fully automatic way in the field. We have proposed such a method, which uses the entropy of the histogram of oriented gradient (HOG) to evaluate the goodness of the compensation. Its performance was satisfactory, but the computational burden was too heavy to be executed by drive assistance devices. In this report, we discuss a method to speed up the algorithm, and obtain a new light algorithm feasible for such devices. We also show more comprehensive performance evaluation results then those in the previous reports.

Barrel distortion is a critical problem that can hinder the successful application of wide-angle cameras. This letter presents an implementation method for fast correction of the barrel distortion. In the proposed method, the required scaling factor is obtained by interpolating a mapping polynomial with a non-uniform spline instead of calculating it directly, which reduces the number of computations required for the distortion correction. This reduction in the number of computations leads to faster correction while maintaining quality: when compared to the conventional method, the reduction ratio of the correction time is about 89%, and the correction quality is 35.3 dB in terms of the average peak signal-to-noise ratio.

In this study, we introduce a system for tracking multiple people using multiple active cameras. Our main objective is to surveille as many targets as possible, at any time, using a limited number of active cameras. In our context, an active camera is a statically located pan-tilt-zoom camera. In this research, we aim to optimize the camera configuration to achieve maximum coverage of the targets. We first devise a method for efficient tracking and estimation of target locations in the environment. Our tracking method is able to track an unknown number of targets and easily estimate multiple future time-steps, which is a requirement for active cameras. Next, we present an optimization of camera configuration with variable time-step that is optimal given the estimated object likelihoods for multiple future frames. We confirmed our results using simulation and real videos, and show that without introducing any significant computational complexities, it is possible to use active cameras to the point that we can track and observe multiple targets very effectively.

Digital still cameras emerged following the introduction of the Sony Mavica analog prototype camera in 1981. These early cameras produced poor image quality and did not challenge film cameras for overall quality. By 1995 digital still cameras in expensive SLR formats had 6 mega-pixels and produced high quality images (with significant image processing). In 2005 significant improvement in image quality was apparent and lower prices for digital still cameras (DSCs) started a rapid decline in film usage and film camera sells. By 2010 film usage was mostly limited to professionals and the motion picture industry. The rise of DSCs was marked by a “pixel war” where the driving feature of the cameras was the pixel count where even moderate cost, ∼ $120, DSCs would have 14 mega-pixels. The improvement of CMOS technology pushed this trend of lower prices and higher pixel counts. Only the single lens reflex cameras had large sensors and large pixels. The drive for smaller pixels hurt the quality aspects of the final image (sharpness, noise, speed, and exposure latitude). Only today are camera manufactures starting to reverse their course and producing DSCs with larger sensors and pixels. This paper will explore why larger pixels and sensors are key to the future of DSCs.

We propose a method for calibrating the topology of distributed pan-tilt cameras (i.e. the structure of routes among and within FOVs) and its probabilistic model. To observe as many objects as possible for as long as possible, pan-tilt control is an important issue in automatic calibration as well as in tracking. In a calibration period, each camera should be controlled towards an object that goes through an unreliable route whose topology is not calibrated yet. This camera control allows us to efficiently establish the topology model. After the topology model is established, the camera should be directed towards the route with the biggest possibility of object observation. We propose a camera control framework based on the mixture of the reliability of the estimated routes and the probability of object observation. This framework is applicable both to camera calibration and object tracking by adjusting weight variables. Experiments demonstrate the efficiency of our camera control scheme for establishing the camera topology model and tracking objects as long as possible.

This paper introduces a tensorial representation of multiple cameras with arbitrary curvilinear motions. It enables us to define a multilinear relationship among image points derived from non-rigid object motions viewed from multiple cameras with arbitrary curvilinear motions. We show the new multilinear relationship is useful for generating images and reconstructing 3D non-rigid object motions viewed from cameras with arbitrary curvilinear motions. The method is tested in real image sequences.

The spatio-temporal multiple view geometry can represent the geometry of multiple images in the case where non-rigid arbitrary motions are viewed from multiple translational cameras. However, it requires many corresponding points and is sensitive to the image noise. In this paper, we investigate mutual projections of cameras in four-dimensional space and show that it enables us to reduce the number of corresponding points required for computing the spatio-temporal multiple view geometry. Surprisingly, take three views for instance, we no longer need any corresponding point to calculate the spatio-temporal multiple view geometry, if all the cameras are projected to the other cameras mutually for two time intervals. We also show that the stability of the computation of spatio-temporal multiple view geometry is drastically improved by considering the mutual projections of cameras.

Recently, many application systems have been developed by using a large number of cameras. If 3D points are observed from synchronized cameras, the multiple view geometry of these cameras can be computed and the 3D reconstruction of the scene is available. Thus, the synchronization of multiple cameras is essential. In this paper, we propose a method for synchronizing multiple cameras and for computing the epipolar geometry from uncalibrated and unsynchronized cameras. In particular we using affine invariance to match the frame numbers of camera images for finding the synchronization. The proposed method is tested by using real image sequences taken from uncalibrated and unsynchronized cameras.

Recently, researches and developments for measuring and modeling of the human body have been receiving much attention. Our aim is to reconstruct an accurate shape of a human foot from multiple camera images, which can capture dynamic behavior of the object. In this paper, a foot-shape database is used for accurate reconstruction of human foot. By using Principal Component Analysis, the foot shape can be represented with new meaningful variables. The dimensionality of the data is also reduced. Thus, the shape of object can be recovered efficiently, even though the object is partially occluded in some input views. To demonstrate the proposed method, two kinds of experiments are presented: reconstruction of human foot in a virtual reality environment with CG multi-camera images, and in real world with eight CCD cameras. In the experiments, the reconstructed shape error with our method is around 2 mm in average, while the error is more than 4 mm with conventional volume intersection method.

The goal of gaze detection is to locate the position (on a monitor) where a user is looking. Previous researches use one wide view camera, which can capture the user's entire face. However, the image resolution is too low with such a camera and the fine movements of user's eye cannot be exactly detected. So, we propose the new gaze detection system with dual cameras (a wide and a narrow view camera). In order to locate the user's eye position accurately, the narrow-view camera has the functionalities of auto focusing/panning/tilting based on the detected 3D eye positions from the wide view camera. In addition, we use the IR-LED illuminators for wide and narrow view camera, which can ease the detecting of facial features, pupil and iris position. To overcome the problem of specular reflection on glasses by illuminator, we use dual IR-LED illuminators for wide and narrow view camera and detect the accurate eye position, which is not hidden by the specular reflection. Experimental results show that the gaze detection error between the computed positions and the real ones is about 2.89 cm of RMS error.

This paper proposes a new method that can robustly recover 3D structure and 3D motion of 3D moving objects from a few multi-views. It recovers 3D feature points by obtaining intersections of back-projection lines which are connected from the camera's optical centers thorough projected feature points on the image planes corresponding to the different cameras. We show that our method needs only six views to suppress false 3D feature points in most cases by discussing the relation between the occurrence probability of false 3D feature points and the number of views. This discussion gives us a criterion to design the optimal multi-camera system for recovering 3D structure and 3D motion of 3D moving objects. An experimental multi-camera system is constructed to confirm the validity of our method. This system can take images from six different views at once and record motion image sequence from each view over a period of a few seconds. It is tested successfully on recovering the 3D structure of Vinus's plaster head and on recovering the 3D structure and 3D motion of a moving hand.