Miniaturization of a coherent monocular structured illumination system for future combination with digital holography

3D-measurement systems based on structured illumination as well as the methods of digital holography have been successfully used in many areas of science and technology for many years^1-5. The advantages of the structured illumination lie in the fast and accurate determination of absolute 3D-object coordinates^6-9, but photogrammetric 3D-measurement techniques are limited in their axial resolution, not surpassing 1 µm in deviation. Digital holography is capable of tracking deviations (displacement vectors) in the range of the used wavelength and below but requires coherent light as illumination source^10-14. A combination of the measurement of absolute position and relative (height) variation seems therefore quite useful. In structured illumination, there have been methods for some years that work with coherent light (e. g. speckles) that can therefore be well linked to digital holography, which in principle has already been demonstrated for stereo camera systems^{15, 16}. However, these combined systems of stereophotogrammetry and digital holography are relatively large and therefore not suitable for many interesting applications in science and technology that require geometrically small and compact measuring heads. These kinds of measurements are usually carried out with miniaturized or endoscopic systems^17-20. For instance, most medical applications must work in a very close range to the test object. On the other hand, similar industrial applications require a certain working distance from measurement instrument to object, such as pick-and-place tasks or product control applications. With this premise in mind a technique has been miniaturized, that was first developed for incoherent Multi-Aperture-Array-Projectors (MAAP)^21-23 and then used with coherent illumination as well²⁴.

In the following text a short description of the basic concepts of 3D-measurement with MAAPs, a recap of the use of repeatable speckles for 3D-measurement and the combination of 3D-measurement and holography is given. We then present our novel findings regarding details about the miniaturization of a 3D-measurement system utilizing repeatable speckles as well as results and a discussion about potentials and drawbacks of the technique.

Specialized cameras are often the priciest components of a stereophotogrammetric setup. This holds true for high-speed measurements^{6, 25} as well as underwater²⁶ or infrared²⁷ applications. Avoiding some of the costs and at the same time reducing the required space by using only a one-camera or monocular setup can therefore be highly advantageous. A monocular system is from our point of view therefore each system making use of only one imaging path. The possibility to reduce a setup in such a way is dependent on the quality of the projector unit which includes the repeatability of the projected patterns. For single aperture projectors such as digital light processing (DLP) projectors, the calibration technique is a well-known practice^28-30. However, these techniques should not be confused with “depth from single image”³¹.

Since single aperture off the shelf projectors are subject to certain speed limitations, Multi-Aperture-Array-Projectors (MAAPs) have been developed, utilizing multiple projection centers and thus allowing a higher projection rate of up to 3 kHz⁶. In the stereophotogrammetric approach to gather 3D-information a correlation search is performed, leading to a correlation coefficient ρ with a typical threshold of 0.9. The correlation coefficient can be calculated by

Here $ {I}_{n},{J}_{n} $ represent the gray values in pixels I and J in each of one of the two cameras, in the $ n $-th image of the measurement sequence. $ \overline{I},\overline{J} $ represent the average gray values over the sequence at the pixels $ I $ and $ J $. These MAAPs have also been used to calibrate a monocular 3D-measurement system with the help of a voxel calibration technique²¹.

Here a linear translation stage has been used to move a reference plane through the measurement volume as depicted in Fig. 1. The patterns projected by the MAAP have been captured separately at every stage position, so that the volume was calibrated voxel-wise. The reconstruction process is based on the comparison of the stack of reference gray values gained from the calibration process and measurement images of the object under test. The same formula (1) could be used to distinguish correlating positions, even though the cross-correlation threshold had to be set to 0.99, as empirically found by Wong et al.²¹ to avoid false reconstructions. The threshold differs from typical values known from stereo photogrammetry (for example 0.9³²) especially due to the different type of correlation search, that is performed in axial direction here. The number of images depends on the number of possible patterns to be generated by the MAAP (e.g., six by Wong et al.²¹). The calibrated system was then used to measure the shape of 3D-objects in high speed and with precise and accurate results. The MAAPs themselves however are as large or even larger than the cameras used and not miniaturizable without extensive effort.

Fig. 1  Sketch of the calibration of a monocular camera system with a Multi-Aperture-Array-Projector (MAAP) for 3D-measurement. The plane can be moved to 100 – 10,000 positions separated by a distance between 1 µm - 100 µm, in dependence on the desired depth and quality of the calibration. At each position the gray values of all the patterns are captured and stored, creating a stack of reference gray values per voxel.

When a coherent beam is statistically scattered areas of constructive and destructive interference are generated, which are called speckles. These patterns can be used as illumination for active stereophotogrammetry³². Apart from the coherent beam, speckle projection itself only requires a scattering element or surface of the size of the beam diameter. By illuminating a diffuser plate with coherent light led through fibers fixed on the diffuser plate the same speckle distribution could be reproduced, as often as the respective fiber was addressed by the coherent beam.

With a fiber switch and multiple fibers, each fixed to a diffuser plate, a number or sequence of such patterns could be repeatedly generated. We successfully applied this technique to generate 3D-reconstructions, using galvanometric mirrors to switch between fibers²⁴. Here the above-described calibration approach for MAAPs was used, resulting in an effective measurement field of 10 cm × 10 cm × 5 cm. An image of a test object (relief of the German physicist Ernst Abbe) and of its 3D-reconstruction is depicted in Fig. 2.

Fig. 2  a Test object, relief of German physicist Ernst Abbe, and b 3D-reconstruction using a macroscopic but monocular speckle projector as used in²⁴. The false coloring represents the z-value of each reconstructed point in µm. Please note that unreconstructed areas have been set to 0.

From these experiments it became clear that the effectively required projector size, derived from the illuminated spots on the diffuser, was much smaller than the size of the used projector (5 cm × 5 cm). A miniaturization was therefore easier to realize compared with an incoherent projector that would rely on optical components for the projection process.

As previously mentioned, 3D-measurement techniques such as active stereophotogrammetry can deliver accurate and precise values of the absolute position of an objects surface, while holography, such as off-axis holography, can detect small relative changes of object surfaces, even below the wavelength in use. The requirement for holographic measurement is the coherence of the light used to illuminate the objects. The scattered light and a reference wave from the same laser source have to overlap, so that interference patterns are generated. The interference patterns can then be used to describe height variations of the surface compared to a master hologram or to the interference fringes induced by strain, pressure, or temperature changes. In earlier works these two techniques, 3D-measurement and off-axis holography, have been combined so that absolute and relative object information could be gathered simultaneously^{15, 16} (see Fig. 3).

Fig. 3  Schematic drawing of the experimental setup used to combine stereophotogrammetric 3D measurement and holographic deformation measurement in a previous paper¹⁶.

Here, a stereophotogrammetric technique has been adapted, that involved coherent structured illumination. A sequence of 30 stereo images was taken, each with a different coherent pattern projected on the object. 3D-points were found by a correlation search between the gray value stacks of the images taken from both measurement cameras. By calculating the correlation coefficient (see Eq. 1) it was possible to find best corresponding pixels in both camera views and hence to reconstruct a 3D-point cloud (Fig. 4a).

Fig. 4  a Image of the reconstructed 3D point cloud of a table tennis ball (values in mm) taken from a previous publication¹⁵. Reprinted/Adapted with permission from [Optics Letters 35(8) (2010)] © The Optical Society. b The interference patterns have been mapped on the point cloud, taken from a previous publication¹⁶. [Reprinted/Adapted] with permission from [Optics Letters 36(23) (2011)] © The Optical Society.

The coherent light scattered by the object has been overlapped with a reference wave from the same laser source on the camera chips, leading to interference patterns in the images, so called off-axis holograms. These patterns changed when the surface under test was deformed even slightly, for example, by temperature variations. The test object was a table tennis ball with a radius of about 18.9 mm, whose reconstructed 3D-point cloud can be seen in Fig. 4a. Depicted in Fig. 4b are interference patterns mapped on the point cloud. Remarkably the system could find deformations with absolute values of up to 1.3 µm.

With the addition of more cameras, it was also possible to calculate the complete deformation vectors in space. The next step to effectively combine the two techniques and enable scenarios in hard accessible volumes where temperature changes and pressure/strain deformations are measurable simultaneously, requires a miniaturization of the 3D-measurement technique. In order to achieve that goal, we next combined optical fibers with a diffuser plate not much larger than the illumination spots to generate repeatable laser speckles and a small camera, calibrated after using the technique from Wong et al.²¹.

The described miniaturization for the speckle generation process²⁴ introduces new proportions to the design of such monocular measurement systems: The limiting factor of the setup size is now, in principle, not the projector itself, but the camera (see Fig. 5).

Fig. 5  Sketch of the miniaturized setup for speckle-based monocular 3D measurement. A fiber coupled laser generates coherent light that is distributed over a fiber switch to different single mode fibers, all connected to the measurement head. To illuminate the object only one fiber is addressed at a time and for a measurement a sequence of images, each with a separate pattern had to be taken. b The measurement head connects the fibers, the diffuser plate and the camera in a stable fashion, so that the same patterns are repeatedly filling the same volumes as seen from the camera.

To miniaturize the setup as much as possible while keeping a stable setup, that allows not only strict reproducibility of speckles but also fixed distances from camera to speckle generation spots and a good beam quality, a fiber switch (Leonie, type: “FO Switch eol 1 × 12 VIS HP”) and a fiber coupled laser (Integrated Optics, type: 0532-13B/532 nm DPSS, MatchBox series 2) have been connected to a specifically designed measurement head (Fig. 6a) with a diameter of 2.6 cm, without fibers. The fibers themselves have been chosen to be single mode fibers (Thorlabs; type: “P3-460B-FC-1”), so that only the intensity of the output and not the profile of the illumination spot would be changed due to twists of the fibers. Additionally, a circular diffuser plate with a diameter of 2.5 cm, containing a centered, circular opening with a diameter of 1.6 cm, was fitted to the measurement head. This diffuser plate was used to scatter the light coming out of the fibers, while the centered camera maintained a clear view of the measurement field. The miniature camera and objective (XIMEA MU9PM-MH, CMOS) was chosen due to its small size (Edmund Optics 7,2 mm FL f/1.8 Red M12) and the relatively small distortion caused in relation to its size and working distance. Ten fibers were arranged and fixed on the measurement head around the camera, using a UV-hardened plastic glue, so that a strict reproducibility of the speckle patterns was possible. This arrangement led to the problem that a change of position of the fibers relative to the diffuser plate required each fiber to be removed and newly connected. The complete measurement head, also including the fibers, with a diameter of now 2.8 cm and a length of about 5 cm is shown in Fig 6b.

Fig. 6  a Photography of the measurement head compared to a 2 € - coin without fibers but with diffuser plate and camera. The openings to lead the single mode fibers can be seen on the sides of the head, leading to the diffuser plate. b The completed head with connected fibers in a three-point holder is depicted. c One of the speckle patterns projected on the reference plane for the calibration.

The working distance was set to be 10 cm, comparable to the working distance used with the 3D-measurement setups utilizing the MAAP^21-23. One of the speckle patterns captured by the camera when projected on the reference plane is shown in Fig. 6c. The calibration was carried out as explained in the basics: A reference plane was mounted on a linear stage and moved through the measurement volume in 500 steps of 100 µm each, resulting in a total measurement volume depth of 5 cm. With the number of calibration steps the amount of stored data and the calculation effort increase. The best trade-off for in the regard of the authors was therefore the here presented measurement volume depth. At each position of the plain an image was captured for every projected set of objective speckles. The stack of reference images was used to find the objects z-position by comparing the measurement images to the gray values stored in the calibration stack as previously shown by Wong et al.²² It has to be remarked that the calibration process itself was not altered or adapted to fit the coherent illumination by objective speckles in difference to the incoherent projection process, that was originally addressed with this technique. The axial size of the speckles had therefore no influence on the calibration process. Only the resulting gray value distribution have been stored and compared. The coherent noise that develops when coherently illuminated surfaces are imaged, so called subjective speckles, were also present in the measurements as well as during the calibration process. A technique to reduce this noise by spatially filtering the images³³ did not lead to an improvement of the reconstruction results.

Most importantly, the main task of the above-described miniaturized projection system was achieved, i.e. the strictly repeatable generation of speckle patterns in order to robustly apply the calibration method mentioned in the basics according to Wong et al. As a proof of concept for possibility of a 3D-measurement with the here proposed miniaturized system, a 3D object (miniature gypsum bust of the German poet Johann Wolfgang von Goethe, height 5 cm, Fig. 7a, a measurement image can be seen in Fig. 7b) could be reconstructed (Fig. 7c). Notably, this result constitutes the first successful implementation of this novel speckle-based 3D measurement with a miniaturized, monocular system. However, the actual reconstruction is relatively imprecise (a ratio between standard deviation and measurement field diagonal of 10⁻³ compared to existing methods, which allow a ratio of 10^{−5 6, 25-27, 32, 33}), but remarkably complete. From the authors' point of view, the reconstruction quality of this first prototype can and will be enhanced to a considerable extent. For instance, there is significant potential for improvement in the structure size of the speckle patterns, which were probably too large in their lateral extension, relative to the intended resolution. A typical value is here a lateral spread of 10 pixel per objective speckles, as seen by the measurement camera³². Due to the comparatively large working distance for an endoscopic approach of 10 cm, only a small angle between the illumination and the observation direction was given, which may have had a negative effect on the point assignment and thus on the 3D-point reconstruction. An additional drawback is coherent noise, so called subjective speckles, that occurs when coherently illuminated surfaces are imaged. The here presented images have also been filtered with different kinds of spatial low pass filters, such as Gaussian and sinusoidal functions. A technique that allowed considerable improvements in speckle-based stereophotogrammetry³³. Unfortunately, this method did only worsen the results to the point, where the bust couldn’t be recognized at all. In the future other reduction methods will have to be found to decrease the influence of this noise. The authors believe that the angle between the illumination and observation and the structure size of the speckles in their present form, are causing the majority of the errors and should be dealt with in a first step of improvement.

Fig. 7  a Photography of a miniature gypsum bust (height 5 cm) of German poet Johann Wolfgang von Goethe. b Single measurement image with the miniaturized speckle based monocular setup. c 3D-reconstruction from 6 patterns. The z-values are depicted in false colors. Please note that unreconstructed points have been depicted with value 0 as well.

In our upcoming work, these parameters shall be investigated and will be varied by designing the basic body of the structure in such a way that adjustments of this kind will be possible without the mechanical disassembly of the device itself. The first step will be an increased number of projected patterns. Previous publication showed that a larger number of patterns reduces the standard deviation³². This will increase the measurement time as well as the effort of reconstruction but should be worth the effort. A decrease in distance from measurement head to object should allow for a larger angle between illumination and observation. Furthermore, the size of the projected speckles has to be optimized be it the number given by³² or away from it.

To the knowledge of the authors the here presented method of 3D-reconstruction has not yet been realized in this way and offers considerable potential for improvement. In addition to these obvious adjustments, however, the use of coherent light also allows the connection of 3D-measurement with holography. If one of the illumination channels or one of the fibers were used to illuminate the camera chip, while the object was illuminated by another fiber at the same time, holograms could also be recorded with this setup. A possible implementation, in this case by off-axis holography, is shown in Fig. 8. Here, a mirror element is used to guid the light of the reference wave to the detector chip. Obviously, holography requires coherent interference between the reference wave and the object wave. To fulfil this condition the fiber generating the reference wave should be longer than the fibers creating speckles to illuminate the object. The path length difference induced in the reference wave fiber, needed to allow reference and object waves to interfere, would depend on the working distance of the device and should at least as long as the light path to the object and back to the detector. In case the coherence length of the laser in use is long enough, path length matching is not mandatory. Also, the intensity of the reference wave has to be adapted to the intensity of the object wave.

Fig. 8  Sketch of a combination of the presented 3D-measurement technique with an off-axis holographic approach.

Such a measuring device could, as already shown in previous publications for a stereophotogrammetric setup^{15, 16}, record height changes in the sub-micrometer range in addition to the 3D-shape. In this way, complex and fast processes, such as the cooling of components or the change in shape of an object under pressure or stress, could be observed and evaluated in real time. And with this device these measurements could be carried out in hardly accessible volumes of interest. Future research projects will therefore be oriented in this direction.

In this publication, a novel, miniaturized, monocular 3D-measurement technology was presented and, in addition to the necessary basics, the voxel-wise calibration of a measurement volume and the repeatability of speckle patterns. First results were presented that prove the feasibility of the principle and its miniaturization. Compared to other 3D-measurement methods, the reconstruction quality can still be significantly improved, especially in terms of precision, which can only be classified in the millimeter range, for example by better adapting the structure size of the speckles in use to the measurement volume and reducing the working distance. Hence the presented system offers the possibility of including holography using coherent light for illumination. This would potentially enable the visualization of smallest height variations, such as caused by changes in heat or load on an object, and their precise assignment to absolute 3D-coordinates even in hard to access measurement volumes. Future work of the authors will focus on improving the method presented here in this direction and on the adaptation of the 3D-measurement method for high-resolution fluorescence microscopy.

We acknowledge the support by the machine shop (M2) of the faculty of physics and astronomy of the university of Jena helping to design and constructing the measurement head. We also gladly acknowledge the funding by the German federal ministry of education and research (BMBF) under the grant number 03ZZ0473.