In computer vision and computer graphics, 3D reconstruction is the process of capturing the shape and appearance of real objects. This process can be accomplished either by active or passive methods.Moons, Theo, Luc Van Gool, and Maarten Vergauwen. "3D reconstruction from multiple images part 1: Principles." Foundations and Trends in Computer Graphics and Vision 4.4 (2010): 287-404. If the model is allowed to change its shape in time, this is referred to as non-rigid or spatio-temporal reconstruction.Zollhöfer, Michael, et al. "Real-time non-rigid reconstruction using an RGB-D camera." ACM Transactions on Graphics 33.4 (2014): 156.
Motivation and applications The research of 3D reconstruction has always been a difficult goal. By Using 3D reconstruction one can determine any object's 3D profile, as well as knowing the 3D coordinate of any point on the profile. The 3D reconstruction of objects is a generally scientific problem and core technology of a wide variety of fields, such as Computer Aided Geometric Design (CAGD), computer graphics, computer animation, computer vision, medical imaging, computational science, virtual reality, digital media, etc. For instance, the lesion information of the patients can be presented in 3D on the computer, which offers a new and accurate approach in diagnosis and thus has vital clinical value. Digital elevation models can be reconstructed using methods such as airborne laser altimetryVosselman, George, and Sander Dijkman. "3D building model reconstruction from point clouds and ground plans." International archives of photogrammetry remote sensing and spatial information sciences 34.3/W4 (2001): 37-44. or synthetic aperture radar.
Active methods
Active methods, i.e. range data methods, given the depth map, reconstruct the 3D profile by numerical approximation approach and build the object in scenario based on model. These methods actively interfere with the reconstructed object, either mechanically or radiometrically using rangefinders, in order to acquire the depth map, e.g. structured light, laser range finder and other active sensing techniques. A simple example of a mechanical method would use a depth gauge to measure a distance to a rotating object put on a turntable. More applicable radiometric methods emit radiance towards the object and then measure its reflected part. Examples range from moving light sources, colored visible light, time-of-flight lasers to microwaves or 3D ultrasound. See 3D scanning for more details.
Passive methods Passive methods of 3D reconstruction do not interfere with the reconstructed object; they only use a sensor to measure the radiance reflected or emitted by the object's surface to infer its 3D structure through image understanding.Buelthoff, Heinrich H., and Alan L. Yuille. "Shape-from-X: Psychophysics and computation ." Fibers' 91, Boston, MA. International Society for Optics and Photonics, 1991. Typically, the sensor is an image sensor in a camera sensitive to visible light and the input to the method is a set of digital images (one, two or more) or video. In this case we talk about image-based reconstruction and the output is a 3D model. By comparison to active methods, passive methods can be applied to a wider range of situations.
Monocular cues methods Monocular cues methods refer to using one or more images from one viewpoint (camera) to proceed to 3D construction. It makes use of 2D characteristics(e.g. Silhouettes, shading and texture) to measure 3D shape, and that's why it is also named Shape-From-X, where X can be silhouettes, shading, texture etc. 3D reconstruction through monocular cues is simple and quick, and only one appropriate digital image is needed thus only one camera is adequate. Technically, it avoids stereo correspondence, which is fairly complex.
Shape-from-shading Due to the analysis of the shade information in the image, by using Lambertian reflectance, the depth of normal information of the object surface is restored to reconstruct.Horn, Berthold KP. "Shape from shading: A method for obtaining the shape of a smooth opaque object from one view." (1970).
Photometric Stereo This approach is more sophisticated than the shape-of-shading method. Images taken in different lighting conditions are used to solve the depth information. It is worth mentioning that more than one image is required by this approach.
Shape-from-texture Suppose such an object with smooth surface covered by replicated texture units, and its projection from 3D to 2D causes distortion and perspective. Distortion and perspective measured in 2D images provide the hint for inversely solving depth of normal information of the object surface.
Machine Learning Based Solutions Machine learning enables learning the correspondence between the subtle features in the input and the respective 3D equivalent. Deep neural networks have shown to be highly effective for 3D reconstruction from a single color image. This works even for non-photorealistic input images such as sketches. Thanks to the high level of accuracy in the reconstructed 3D features, deep learning based method has been employed for biomedical engineering applications to reconstruct CT imagery from X-ray.
Stereo vision
Stereo vision obtains the 3-dimensional geometric information of an object from multiple images based on the research of human visual system. The results are presented in form of depth maps. Images of an object acquired by two cameras simultaneously in different viewing angles, or by one single camera at different time in different viewing angles, are used to restore its 3D geometric information and reconstruct its 3D profile and location. This is more direct than Monocular methods such as shape-from-shading.
Binocular stereo vision method requires two identical cameras with parallel optical axis to observe one same object, acquiring two images from different points of view. In terms of trigonometry relations, depth information can be calculated from disparity. Binocular stereo vision method is well developed and stably contributes to favorable 3D reconstruction, leading to a better performance when compared to other 3D construction. Unfortunately, it is computationally intensive, besides it performs rather poorly when baseline distance is large.
Problem statement and basics The approach of using Binocular stereo vision to acquire object's 3D geometric information is on the basis of visual disparity.McCoun, Jacques, and Lucien Reeves. Binocular vision: development, depth perception and disorders. Nova Science Publishers, Inc., 2010. The following picture provides a simple schematic diagram of horizontally sighted Binocular Stereo Vision, where b is the baseline between projective centers of two cameras.
The origin of the camera's coordinate system is at the optical center of the camera's lens as shown in the figure. Actually, the camera's image plane is behind the optical center of the camera's lens. However, to simplify the calculation, images are drawn in front of the optical center of the lens by f. The u-axis and v-axis of the image's coordinate system are in the same direction with x-axis and y-axis of the camera's coordinate system respectively. The origin of the image's coordinate system is located on the intersection of imaging plane and the optical axis. Suppose such world point whose corresponding image points are and respectively on the left and right image plane. Assume two cameras are in the same plane, then y-coordinates of and are identical, i.e.,. According to trigonometry relations,
where are coordinates of in the left camera's coordinate system, is focal length of the camera. Visual disparity is defined as the difference in image point location of a certain world point acquired by two cameras,
based on which the coordinates of can be worked out.
Therefore, once the coordinates of image points is known, besides the parameters of two cameras, the 3D coordinate of the point can be determined.
The 3D reconstruction consists of the following sections:
Image acquisition 2D digital image acquisition is the information source of 3D reconstruction. Commonly used 3D reconstruction is based on two or more images, although it may employ only one image in some cases. There are various types of methods for image acquisition that depends on the occasions and purposes of the specific application. Not only the requirements of the application must be met, but also the visual disparity, illumination, performance of camera and the feature of scenario should be considered.
Camera calibration
Camera calibration in Binocular Stereo Vision refers to the determination of the mapping relationship between the image points and , and space coordinate in the 3D scenario. Camera calibration is a basic and essential part in 3D reconstruction via Binocular Stereo Vision.
Feature extraction
The aim of feature extraction is to gain the characteristics of the images, through which the stereo correspondence processes. As a result, the characteristics of the images closely link to the choice of matching methods. There is no such universally applicable theory for features extraction, leading to a great diversity of stereo correspondence in Binocular Stereo Vision research.
Stereo correspondence
Stereo correspondence is to establish the correspondence between primitive factors in images, i.e. to match and from two images. Certain interference factors in the scenario should be noticed, e.g. illumination, noise, surface physical characteristic, etc.
Restoration According to precise correspondence, combined with camera location parameters, 3D geometric information can be recovered without difficulties. Due to the fact that accuracy of 3D reconstruction depends on the precision of correspondence, error of camera location parameters and so on, the previous procedures must be done carefully to achieve relatively accurate 3D reconstruction.
3D Reconstruction of medical images Clinical routine of diagnosis, patient follow-up, computer assisted surgery, surgical planning etc. are facilitated by accurate 3D models of the desired part of human anatomy. Main motivation behind 3D reconstruction includes Improved accuracy due to multi view aggregation. Detailed surface estimates. Can be used to plan, simulate, guide, or otherwise assist a surgeon in performing a medical procedure. The precise position and orientation of the patient's anatomy can be determined. Helps in a number of clinical areas, such as radiotherapy planning and treatment verification, spinal surgery, hip replacement, neurointerventions and aortic stenting. Applications:
3D reconstruction has applications in many fields. They include: Pavement engineering Medicine Free-viewpoint video reconstructionCarranza, Joel, et al. "Free-viewpoint video of human actors." ACM Transactions on Graphics. Vol. 22. No. 3. ACM, 2003. Robotic mappingThrun, Sebastian. "Robotic mapping: A survey." Exploring artificial intelligence in the new millennium 1.1-35 (2002): 1. City planning Tomographic reconstructionXu, Fang, and Klaus Mueller. "Real-time 3D computed tomographic reconstruction using commodity graphics hardware ." Physics in Medicine & Biology 52.12 (2007): 3405. GamingMortara, Michela, et al. "Learning cultural heritage by serious games." Journal of Cultural Heritage 15.3 (2014): 318-325. Virtual environments and virtual tourism Earth observation Archaeology Augmented realityIzadi, Shahram, et al. "KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera." Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 2011. Reverse engineering Motion capture Digital Preservation of Cultural HeritageLeonardo Gomes, Olga Regina Pereira Bellon, Luciano Silva. "3D reconstruction methods for digital preservation of cultural heritage: A survey." Pattern Recognition Letters 50 (2014): 3-14.