Citation:

Holography, and the future of 3D display


  • Light: Advanced Manufacturing  2, Article number: 28 (2021)
More Information
  • Corresponding author:

    Pierre-Alexandre Blanche (pablanche@optics.arizona.edu)

  • Received: 06 May 2021
    Revised: 05 November 2021
    Accepted: 19 November 2021
    Accepted article preview online: 25 November 2021
    Published online: 20 December 2021

doi: https://doi.org/10.37188/lam.2021.028

  • The pioneers of holography, Gabor, Leith, Upatnieks, and Denisyuk, predicted very early that the ultimate 3D display will be based on this technique. This conviction was rooted on the fact that holography is the only approach that can render all optical cues interpreted by the human visual system. Holographic 3D displays have been a dream chased after for many years, facing challenges on all fronts: computation, transmission, and rendering. With numbers such as 6.6 × 1015 flops required for calculations, 3 × 1015 b/s data rates, and 1.6 × 1012 phase pixels, the task has been daunting. This article is reviewing the recent accomplishments made in the field of holographic 3D display. Specifically, the new developments in machine learning and neural network algorithms demonstrating that computer-generated holograms approach real-time processing. A section also discuss the problem of data transmission that can arguably be solved using clever compression algorithms and optical fiber transmission lines. Finally, we introduce the last obstacle to holographic 3D display, which is is the rendering hardware. However, there is no further mystery. With larger and faster spatial light modulators (SLMs), holographic projection systems are constantly improving. The pixel count on liquid crystal on silicon (LCoS) as well as microelectromechanical systems (MEMS) phase displays is increasing by the millions, and new photonic integrated circuit phased arrays are achieving real progress. It is only a matter of time for these systems to leave the laboratory and enter the consumer world. The future of 3D displays is holographic, and it is happening now.
  • 加载中
  • [1] Leith, E. N. & Upatnieks, J. New techniques in wavefront reconstruction. Journal of the Optical Society of America 51, 1469-1473 (1961).
    [2] Denisyuk, Y. N. On the reflection of optical properties of an object in a wave field of light scattered by it. Doklady Akademii Nauk SSSR 144, 1275-1278 (1962).
    [3] De Bitetto, D. A holographic motion picture film with constant velocity transport. Applied Physics Letters 12, 295-297 (1968). doi: 10.1063/1.1651998
    [4] Jacobson, A. D., Evtuhov, V. & Neeland, J. K. Motion picture holography. Applied Physics Letters 14, 120-122 (1969). doi: 10.1063/1.1652740
    [5] Blanche, P. A. Introduction to holographic principles. in Optical Holography: Materials, Theory and Applications (ed Blanche, P. A.) Ch. 1 (Amsterdam: Elsevier, 2020), 1-39.
    [6] Ciciora, W. et al. Cable telephony. in Modern Cable Television Technology: Video, Voice, and Data Communications 2nd edn (eds Ciciora, W. et al) Ch. 6 (Amsterdam: Morgan Kaufmann, 2004), 229-284.
    [7] Plonus, M. Digital systems. in Electronics and Communications for Scientists and Engineers 2nd edn (ed Plonus, M.) Ch. 9 (Oxford: Butterworth-Heinemann, 2020), 355-480.
    [8] Pritchard, D. H. U.S. color television fundamentals: a review. SMPTE Journal 86, 819-828 (1977). doi: 10.5594/J06718
    [9] Dunn, D. et al. Stimulating the human visual system beyond real world performance in future augmented reality displays. Proceedings of 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Porto de Galinhas, Brazil: IEEE, 2020, 90-100.
    [10] Howard, I. P. Perceiving in Depth, Volume 1: Basic Mechanisms. (Oxford: Oxford University Press, 2012).
    [11] Park, M. C. & Mun, S. Overview of measurement methods for factors affecting the human visual system in 3D displays. Journal of Display Technology 11, 877-888 (2015). doi: 10.1109/JDT.2015.2389212
    [12] Mayo, H. Outlines of Human Physiology. 4th edn. (London: Renshaw, 1837).
    [13] Cakmakci, O. & Rolland, J. Head-worn displays: a review. Journal of Display Technology 2, 199-216 (2006). doi: 10.1109/JDT.2006.879846
    [14] Urey, H. et al. State of the art in stereoscopic and autostereoscopic displays. Proceedings of the IEEE 99, 540-555 (2011). doi: 10.1109/JPROC.2010.2098351
    [15] Woods, A. J. Crosstalk in stereoscopic displays: a review. Journal of Electronic Imaging 21, 040902 (2012). doi: 10.1117/1.JEI.21.4.040902
    [16] Read, J. C. A. & Bohr, I. User experience while viewing stereoscopic 3D television. Ergonomics 57, 1140-1153 (2014). doi: 10.1080/00140139.2014.914581
    [17] Sexton, I. & Surman, P. Stereoscopic and autostereoscopic display systems. IEEE Signal Processing Magazine 16, 85-99 (1999). doi: 10.1109/79.768575
    [18] Dodgson, N. A. Autostereoscopic 3D displays. Computer 38, 31-36 (2005). doi: 10.1109/MC.2005.252
    [19] Iizuka, K. Engineering Optics. 3rd edn. (New York: Springer, 2008).
    [20] Rotter, P. Why did the 3D revolution fail?: the present and future of stereoscopy [commentary]. IEEE Technology and Society Magazine 36, 81-85 (2017).
    [21] Chen, Y. S. et al. Video-based eye tracking autostereoscopic displays. Optical Engineering 40, 2726-2734 (2001). doi: 10.1117/1.1416130
    [22] Yi, S. Y., Chae, H. B. & Lee, S. H. Moving parallax barrier design for eye-tracking autostereoscopic displays. Proceedings of 2008 3DTV Conference: the True Vision-Capture, Transmission and Display of 3D Video. Istanbul, Turkey: IEEE, 2008, 165-168.
    [23] Yoon, K. H. et al. Autostereoscopic 3D display system with dynamic fusion of the viewing zone under eye tracking: principles, setup, and evaluation [Invited]. Applied Optics 57, A101-A117 (2018). doi: 10.1364/AO.57.00A101
    [24] Yu, X. B. et al. Large viewing angle three-dimensional display with smooth motion parallax and accurate depth cues. Optics Express 23, 25950-25958 (2015). doi: 10.1364/OE.23.025950
    [25] Kara, P. A. et al. The key performance indicators of projection-based light field visualization. Journal of Information Display 20, 81-93 (2019). doi: 10.1080/15980316.2019.1606120
    [26] Kara, P. A. et al. The interdependence of spatial and angular resolution in the quality of experience of light field visualization. Proceedings of 2017 International Conference on 3D Immersion (IC3D). Brussels, Belgium: IEEE, 2017, 1-8.
    [27] Stern, A., Yitzhaky, Y. & Javidi, B. Perceivable light fields: matching the requirements between the human visual system and autostereoscopic 3-D displays. Proceedings of the IEEE 102, 1571-1587 (2014). doi: 10.1109/JPROC.2014.2348938
    [28] Watanabe, Y. & Kakeya, H. A full-HD super-multiview display based on adaptive time-division multiplexing parallax barrier. ITE Transactions on Media Technology and Applications 8, 230-237 (2020). doi: 10.3169/mta.8.230
    [29] Wan, W. Q. et al. Super multi-view display based on pixelated nanogratings under an illumination of a point light source. Optics and Lasers in Engineering 134, 106258 (2020). doi: 10.1016/j.optlaseng.2020.106258
    [30] Wu, G. C. et al. Light field image processing: an overview. IEEE Journal of Selected Topics in Signal Processing 11, 926-954 (2017). doi: 10.1109/JSTSP.2017.2747126
    [31] Martínez-Corral, M. & Javidi, B. Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems. Advances in Optics and Photonics 10, 512-566 (2018). doi: 10.1364/AOP.10.000512
    [32] Huang, H. K. & Hua, H. High-performance integral-imaging- based light field augmented reality display using freeform optics. Optics Express 26, 17578-17590 (2018). doi: 10.1364/OE.26.017578
    [33] Rogers, B. & Graham, M. Similarities between motion parallax and stereopsis in human depth perception. Vision Research 22, 261-270 (1982). doi: 10.1016/0042-6989(82)90126-2
    [34] Blanche, P. A. et al. Holographic three-dimensional telepresence using large-area photorefractive polymer. Nature 468, 80-83 (2010). doi: 10.1038/nature09521
    [35] Yang, L. et al. Demonstration of a large-size horizontal light-field display based on the LED panel and the micro-pinhole unit array. Optics Communications 414, 140-145 (2018). doi: 10.1016/j.optcom.2017.12.069
    [36] Yu, X. B. et al. Dynamic three-dimensional light-field display with large viewing angle based on compound lenticular lens array and multi-projectors. Optics Express 27, 16024-16031 (2019). doi: 10.1364/OE.27.016024
    [37] Liu, B. Y. et al. Time-multiplexed light field display with 120-degree wide viewing angle. Optics Express 27, 35728-35739 (2019). doi: 10.1364/OE.27.035728
    [38] Gao, X. et al. Full-parallax 3D light field display with uniform view density along the horizontal and vertical direction. Optics Communications 467, 125765 (2020). doi: 10.1016/j.optcom.2020.125765
    [39] Kara, P. A. et al. Cinema as large as life: large-scale light field cinema system. Proceedings of 2017 International Conference on 3D Immersion (IC3D). Brussels, Belgium: IEEE, 2017, 1-8.
    [40] Wang, P. R. et al. A full-parallax tabletop three dimensional light-field display with high viewpoint density and large viewing angle based on space-multiplexed voxel screen. Optics Communications 488, 126757 (2021). doi: 10.1016/j.optcom.2021.126757
    [41] Looking Glass Factory. (2021). at https://lookingglassfactory.com/.
    [42] Lambooij, M. T., IJsselsteijn, W. A. & Heynderickx, I. Visual discomfort in stereoscopic displays: a review. Proceedings of SPIE 6490, Stereoscopic Displays and Virtual Reality Systems XIV. San Jose, CA, United States: SPIE, 2007, 64900I.
    [43] Hoffman, D. M. et al. Vergence–accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 33 (2008).
    [44] Kim, J., Kane, D. & Banks, M. S. The rate of change of vergence–accommodation conflict affects visual discomfort. Vision Research 105, 159-165 (2014). doi: 10.1016/j.visres.2014.10.021
    [45] Daniel, F. & Kapoula, Z. Induced vergence-accommodation conflict reduces cognitive performance in the Stroop test. Scientific Reports 9, 1247 (2019). doi: 10.1038/s41598-018-37778-y
    [46] Watanabe, Y. & Kakeya, H. Time-division and color multiplexing light-field display using liquid-crystal display panels to induce focal accommodation. Applied Optics 60, 1966-1972 (2021). doi: 10.1364/AO.413352
    [47] Li, T. T. et al. View-dependent light-field display that supports accommodation using a commercially-available high pixel density LCD panel. SID Symposium Digest of Technical Papers 51, 1013-1016 (2020). doi: 10.1002/sdtp.14044
    [48] St Hilaire, P. et al. Are stereograms holograms? A human perception analysis of sampled perspective holography. Journal of Physics: Conference Series 415, 012035 (2013). doi: 10.1088/1742-6596/415/1/012035
    [49] Huang, H. K. & Hua, H. Effects of ray position sampling on the visual responses of 3D light field displays. Optics Express 27, 9343-9360 (2019). doi: 10.1364/OE.27.009343
    [50] Zhan, T. et al. Multifocal displays: review and prospect. PhotoniX 1, 10 (2020). doi: 10.1186/s43074-020-00010-0
    [51] Balram, N. & Tošic, I. Light-field imaging and display systems. Information Display 32, 6-13 (2016).
    [52] Jo, Y. et al. Tomographic projector: large scale volumetric display with uniform viewing experiences. ACM Transactions on Graphics 38, 21 (2019).
    [53] Chang, J. H. R., Kumar, B. V. K. V. & Sankaranarayanan, A. C. Towards multifocal displays with dense focal stacks. ACM Transactions on Graphics 37, 198 (2018).
    [54] Smalley, D. et al. Volumetric displays: turning 3-D inside-out. Optics and Photonics News 29, 26-33 (2018).
    [55] Smalley, D. E. et al. A photophoretic-trap volumetric display. Nature 553, 486-490 (2018). doi: 10.1038/nature25176
    [56] Jones, A. et al. Rendering for an interactive 360° light field display. ACM Transactions on Graphics 26, 40-es (2007). doi: 10.1145/1276377.1276427
    [57] Hirayama, R. et al. A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575, 320-323 (2019). doi: 10.1038/s41586-019-1739-5
    [58] Kumagai, K., Hasegawa, S. & Hayasaki, Y. Volumetric bubble display. Optica 4, 298-302 (2017). doi: 10.1364/OPTICA.4.000298
    [59] Rogers, W. & Smalley, D. Simulating virtual images in optical trap displays. Scientific Reports 11, 7522 (2021). doi: 10.1038/s41598-021-86495-6
    [60] Suzuki, K., Fukano, Y. & Oku, H. 1000-volume/s high-speed volumetric display for high-speed HMD. Optics Express 28, 29455-29468 (2020). doi: 10.1364/OE.401778
    [61] Baek, H. et al. Wheel screen type lamina 3D display system with enhanced resolution. Current Optics and Photonics 5, 23-31 (2021).
    [62] Sarakinos, A. & Lembessis, A. Color holography for the documentation and dissemination of cultural heritage: optoclonesTM from four museums in two countries. Journal of Imaging 5, 59 (2019). doi: 10.3390/jimaging5060059
    [63] Gentet, P. et al. Evaluation of the realism of a full-color reflection H2 analog hologram recorded on ultra-fine-grain silver-halide material. Open Physics 17, 449-457 (2019). doi: 10.1515/phys-2019-0046
    [64] Lohmann, A. W. & Paris, D. P. Binary fraunhofer holograms, generated by computer. Applied Optics 6, 1739-1748 (1967). doi: 10.1364/AO.6.001739
    [65] Lesem, L. B, Hirsch, P. M. & Jordan, J. A. The kinoform: a new wavefront reconstruction device. IBM Journal of Research and Development 13, 150-155 (1969). doi: 10.1147/rd.132.0150
    [66] Brown, B. R. & Lohmann, A. W. Computer-generated binary holograms. IBM Journal of research and Development 13, 160-168 (1969). doi: 10.1147/rd.132.0160
    [67] Cameron, C. D. et al. Computational challenges of emerging novel true 3D holographic displays. Proceedings of SPIE 4109, Critical Technologies for the Future of Computing. San Diego, CA, United States: SPIE, 2000, 129-140.
    [68] Slinger, C., Cameron, C. & Stanley, M. Computer-generated holography as a generic display technology. Computer 38, 46-53 (2005).
    [69] Cooley, J. W. & Tukey, J. W. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation 19, 297-301 (1965). doi: 10.1090/S0025-5718-1965-0178586-1
    [70] Bianco, V. et al. Quasi noise-free digital holography. Light: Science & Applications 5, e16142 (2016).
    [71] Gerchberg, R. W. & Saxton, W. O. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35, 237-250 (1972).
    [72] Makowski, M. et al. Iterative design of multiplane holograms: experiments and applications. Optical Engineering 46, 045802 (2007). doi: 10.1117/1.2727379
    [73] Makowski, M., Sypek, M. & Kolodziejczyk, A. Colorful reconstructions from a thin multi-plane phase hologram. Optics Express 16, 11618-11623 (2008). doi: 10.1364/OE.16.011618
    [74] Velez-Zea, A. Iterative multiplane hologram generation with mixed constraint. Applied Optics 60, 224-231 (2021). doi: 10.1364/AO.408402
    [75] Zhang, H., Cao, L. C. & Jin, G. F. Computer-generated hologram with occlusion effect using layer-based processing. Applied Optics 56, F138-F143 (2017). doi: 10.1364/AO.56.00F138
    [76] Liu, J. P. & Liao, H. K. Fast occlusion processing for a polygon-based computer-generated hologram using the slice-by-slice silhouette method. Applied Optics 57, A215-A221 (2018). doi: 10.1364/AO.57.00A215
    [77] Liu, S. T. et al. Occlusion calculation algorithm for computer generated hologram based on ray tracing. Optics Communications 443, 76-85 (2019). doi: 10.1016/j.optcom.2019.03.007
    [78] Sahin, E. et al. Computer-generated holograms for 3D imaging: a survey. ACM Computing Surveys 53, 32 (2020).
    [79] Su, Y. F. et al. Projection-type multiview holographic three-dimensional display using a single spatial light modulator and a directional diffractive device. IEEE Photonics Journal 10, 7000512 (2018).
    [80] Wang, Z. et al. Resolution-enhanced holographic stereogram based on integral imaging using moving array lenslet technique. Applied Physics Letters 113, 221109 (2018). doi: 10.1063/1.5063273
    [81] Zhang, X. et al. Resolution-enhanced holographic stereogram based on integral imaging using an intermediate-view synthesis technique. Optics Communications 457, 124656 (2020). doi: 10.1016/j.optcom.2019.124656
    [82] Shi, L. et al. Towards real-time photorealistic 3d holography with deep neural networks. Nature 591, 234-239 (2021). doi: 10.1038/s41586-020-03152-0
    [83] Nishitsuji, T. et al. Review of fast calculation techniques for computer-generated holograms with the point-light-source-based model. IEEE Transactions on Industrial Informatics 13, 2447-2454 (2017). doi: 10.1109/TII.2017.2669200
    [84] Tsang, P. W. M., Poon, T. C. & Wu, Y. M. Review of fast methods for point-based computer-generated holography [Invited]. Photonics Research 6, 837-846 (2018). doi: 10.1364/PRJ.6.000837
    [85] Lucente, M. E. Interactive computation of holograms using a look-up table. Journal of Electronic Imaging 2, 28-34 (1993). doi: 10.1117/12.133376
    [86] Sato, H. et al. Real-time colour hologram generation based on ray-sampling plane with multi-GPU acceleration. Scientific Reports 8, 1500 (2018). doi: 10.1038/s41598-018-19361-7
    [87] Pi, D. P. et al. Acceleration of computer-generated hologram using wavefront-recording plane and look-up table in three-dimensional holographic display. Optics Express 28, 9833-9841 (2020). doi: 10.1364/OE.385388
    [88] Wang, Z. et al. Simple and fast calculation algorithm for computer-generated hologram based on integral imaging using look-up table. Optics Express 26, 13322-13330 (2018). doi: 10.1364/OE.26.013322
    [89] Wang, Y. C. et al. Hardware implementations of computer-generated holography: a review. Optical Engineering 59, 102413 (2020).
    [90] Sugie, T. et al. High-performance parallel computing for next-generation holographic imaging. Nature Electronics 1, 254-259 (2018). doi: 10.1038/s41928-018-0057-5
    [91] Shimobaba, T. & Ito, T. Computer Holography: Acceleration Algorithms and Hardware Implementations. (Boca Raton: CRC Press, 2019).
    [92] Yu, C. Y. et al. Four-image encryption scheme based on quaternion Fresnel transform, chaos and computer generated hologram. Multimedia Tools and Applications 77, 4585-4608 (2018). doi: 10.1007/s11042-017-4637-6
    [93] Sun, M. Y. et al. Acceleration and expansion of a photorealistic computer-generated hologram using backward ray tracing and multiple off-axis wavefront recording plane methods. Optics Express 28, 34994-35005 (2020). doi: 10.1364/OE.410314
    [94] Desiderio, K. & Phillips, I. How Pixar’s animation has evolved over 24 years, from ‘toy story’ to ‘toy story 4’. (2019). at https://blog.adafruit.com/2019/06/30/how-pixars-animation-has-evolved-over-24-years-from-toy-story-to-toy-story-4/.
    [95] Horisaki, R., Takagi, R. & Tanida, J. Deep-learning-generated holography. Applied Optics 57, 3859-3863 (2018). doi: 10.1364/AO.57.003859
    [96] Lindsay, M. et al. Machine learning assisted holography. Proceedings of SPIE 11731, Computational Imaging VI. SPIE, 2021, 1173103.
    [97] Peng, Y. F. et al. Neural holography with camera-in-the-loop training. ACM Transactions on Graphics 39, 185 (2020).
    [98] Chakravarthula, P. et al. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics 39, 186 (2020).
    [99] Lee, J. et al. Deep neural network for multi-depth hologram generation and its training strategy. Optics Express 28, 27137-27154 (2020). doi: 10.1364/OE.402317
    [100] Wu, J. C. et al. High-speed computer-generated holography using an autoencoder-based deep neural network. Optics Letters 46, 2908-2911 (2021). doi: 10.1364/OL.425485
    [101] Yanagihara, H. et al. Real-time three-dimensional video reconstruction of real scenes with deep depth using electro-holographic display system. Optics Express 27, 15662-15678 (2019). doi: 10.1364/OE.27.015662
    [102] Zhao, Y. et al. Fast calculation method of computer-generated hologram using a depth camera with point cloud gridding. Optics Communications 411, 166-169 (2018). doi: 10.1016/j.optcom.2017.11.040
    [103] Sofana, R. S. et al. Future generation 5G wireless networks for smart grid: a comprehensive review. Energies 12, 2140 (2019). doi: 10.3390/en12112140
    [104] Bohli, A. & Bouallegue, R. How to meet increased capacities by future green 5G networks: a survey. IEEE Access 7, 42220-42237 (2019). doi: 10.1109/ACCESS.2019.2907284
    [105] El Rhammad, A. et al. Color digital hologram compression based on matching pursuit. Applied Optics 57, 4930-4942 (2018). doi: 10.1364/AO.57.004930
    [106] Jiao, S. M. et al. Compression of phase-only holograms with JPEG standard and deep learning. Applied Sciences 8, 1258 (2018). doi: 10.3390/app8081258
    [107] Stepień, P. et al. Hologram compression in quantitative phase imaging. Proceedings of SPIE 11249, Quantitative Phase Imaging VI. San Francisco, California, United States: SPIE, 2020, 112491Q.
    [108] Liu, M. J., Yang, G. L. & Xie, H. Y. Method of computer-generated hologram compression and transmission using quantum back-propagation neural network. Optical Engineering 56, 023104 (2017). doi: 10.1117/1.OE.56.2.023104
    [109] Zeng, Z. X. et al. Full-color holographic display with increased-viewing-angle [Invited]. Applied Optics 56, F112-F120 (2017). doi: 10.1364/AO.56.00F112
    [110] Lum, Z. M. A. et al. Increasing pixel count of holograms for three-dimensional holographic display by optical scan-tiling. Optical Engineering 52, 015802 (2013). doi: 10.1117/1.OE.52.1.015802
    [111] Onural, L., Yaraş, F. & Kang, H. Digital holographic three-dimensional video displays. Proceedings of the IEEE 99, 576-589 (2011). doi: 10.1109/JPROC.2010.2098430
    [112] An, J. et al. Slim-panel holographic video display. Nature Communications 11, 5568 (2020). doi: 10.1038/s41467-020-19298-4
    [113] Takaki, Y. & Yokouchi, M. Accommodation measurements of horizontally scanning holographic display. Optics Express 20, 3918-3931 (2012). doi: 10.1364/OE.20.003918
    [114] Kozacki, T. et al. Fourier horizontal parallax only computer and digital holography of large size. Optics Express 29, 18173-18191 (2021). doi: 10.1364/OE.421186
    [115] Henrie, A. et al. Hardware and software improvements to a low-cost horizontal parallax holographic video monitor. Applied Optics 57, A122-A133 (2018). doi: 10.1364/AO.57.00A122
    [116] Sando, Y. et al. Real-time interactive holographic 3D display with a 360° horizontal viewing zone. Applied Optics 58, G1-G5 (2019). doi: 10.1364/AO.58.0000G1
    [117] Maimone, A., Georgiou, A. & Kollin, J. S. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics 36, 85 (2017).
    [118] Duan, X. H. et al. Full-color see-through near-eye holographic display with 80° field of view and an expanded eye-box. Optics Express 28, 31316-31329 (2020). doi: 10.1364/OE.399359
    [119] Zaperty, W., Makowski, P. L. & Kozacki, T. Multi-SLM color holographic 3D display of real-world holographic content with numerical data adaptation. Frontiers in Optics 2017. Washington, D.C. United States: Optical Society of America, 2017, FTu4C.3.
    [120] Leister, N. et al. Full-color interactive holographic projection system for large 3D scene reconstruction. Proceedings of SPIE 6911, Emerging Liquid Crystal Technologies III. San Jose, California, United States: SPIE, 2008, 202-211.
    [121] Häussler, R., Leister, N. & Stolle, H. Large holographic 3D display for real-time computer-generated holography. Proceedings of SPIE 10335, Digital Optical Technologies 2017. Munich, Germany: SPIE, 2017, 177-187.
    [122] Lazarev, G. et al. LCOS spatial light modulators: trends and applications. in Optical Imaging and Metrology: Advanced Technologies (eds Osten, W. & Reingand, N.) Ch. 1 (Weinheim: Wiley, 2012), 1-29.
    [123] Wang, Y. M. et al. 2D broadband beamsteering with large-scale MEMS optical phased array. Optica 6, 557-562 (2019). doi: 10.1364/OPTICA.6.000557
    [124] Dudley, D., Duncan, W. M. & Slaughter, J. Emerging digital micromirror device (DMD) applications. Proceedings of SPIE 4985, MOEMS Display and Imaging Systems. San Jose, CA, United States: SPIE, 2003, 14-25.
    [125] Chlipala, M. & Kozacki, T. Color LED DMD holographic display with high resolution across large depth. Optics Letters 44, 4255-4258 (2019). doi: 10.1364/OL.44.004255
    [126] Son, J. Y. et al. Holographic display based on a spatial DMD array. Optics Letters 38, 3173-3176 (2013). doi: 10.1364/OL.38.003173
    [127] Jiao, S. M. et al. Complex-amplitude holographic projection with a digital micromirror device (DMD) and error diffusion algorithm. IEEE Journal of Selected Topics in Quantum Electronics 26, 2800108 (2020).
    [128] Bloom, D. M. Grating light valve: revolutionizing display technology. Proceedings of SPIE 3013, Projection Displays III. San Jose, CA, United States: SPIE, 1997, 165-171.
    [129] Saruta, K. et al. Nanometer-order control of MEMS ribbons for blazed grating light valves. Proceedings of the 19th IEEE International Conference on Micro Electro Mechanical Systems. Istanbul, Turkey: IEEE, 2006, 842-845.
    [130] Lin, T. H. Implementation and characterization of a flexure-beam micromechanical spatial light modulator. Optical Engineering 33, 3643-3648 (1994). doi: 10.1117/12.181578
    [131] Douglass, M. DMD reliability: a MEMS success story. Proceedings of SPIE 4980, Reliability, Testing, and Characterization of MEMS/MOEMS II. San Jose, CA, United States: SPIE, 2003, 1-11.
    [132] Monk, D. W. Digital light processing: a new image technology for the television of the future. 1997 International Broadcasting Convention IBS 97. Amsterdam, Netherlands: IET, 1997, 581-586.
    [133] Bartlett, T. A., McDonald, W. C. & Hall, J. N. Adapting Texas instruments DLP technology to demonstrate a phase spatial light modulator. Proceedings of SPIE 10932, Emerging Digital Micromirror Device Based Systems and Applications XI. San Francisco, California, United States: SPIE, 2019, 109320S.
    [134] Bartlett, T. A. et al. Recent advances in the development of the Texas instruments phase-only microelectromechanical systems (MEMS) spatial light modulator. Proceedings of SPIE 11698, Emerging Digital Micromirror Device Based Systems and Applications XIII. SPIE, 2021, 116980O.
    [135] Ketchum, R. S. & Blanche, P. A. Diffraction efficiency characteristics for MEMS-based phase-only spatial light modulator with nonlinear phase distribution. Photonics 8, 62 (2021). doi: 10.3390/photonics8030062
    [136] Blanche, P. A. Photorefractive Organic Materials and Applications. (Cham: Springer, 2016).
    [137] Li, X. et al. Highly photorefractive hybrid liquid crystal device for a video-rate holographic display. Optics Express 24, 8824-8831 (2016). doi: 10.1364/OE.24.008824
    [138] Benton, S. A. Experiments in holographic video imaging. Proceedings of SPIE 10308, Holography. Tatabánya, Hungary: SPIE, 1990, 103080C.
    [139] St-Hilaire, P. et al. Color images with the MIT holographic video display. Proceedings of SPIE 1667, Practical Holography VI. San Jose, CA, United States: SPIE, 1992, 73-84.
    [140] Matteo, A. M., Tsai, C. S. & Do, N. Collinear guided wave to leaky wave acoustooptic interactions in proton-exchanged LiNbO3 waveguides. IEEE Transactions on Ultrasonics,Ferroelectrics,and Frequency Control 47, 16-28 (2000). doi: 10.1109/58.818745
    [141] Smalley, D. E. et al. Anisotropic leaky-mode modulator for holographic video displays. Nature 498, 313-317 (2013). doi: 10.1038/nature12217
    [142] Qaderi, K. & Smalley, D. E. Leaky-mode waveguide modulators with high deflection angle for use in holographic video displays. Optics Express 24, 20831-20841 (2016). doi: 10.1364/OE.24.020831
    [143] Jolly, S. et al. Progress in transparent flat-panel holographic displays enabled by guided-wave acousto-optics. Proceedings of SPIE 10558, Practical Holography XXXII: Displays, Materials, and Applications. San Francisco, California, United States: SPIE, 2018, 105580L.
    [144] Bove, V. M. Jr. Holographic television. in Optical Holography: Materials, Theory and Applications (ed Blanche, P. A.) Ch. 4 (Amsterdam: Elsevier, 2020), 73-82.
    [145] Sun, J. et al. Large-scale nanophotonic phased array. Nature 493, 195-199 (2013). doi: 10.1038/nature11727
    [146] Sun, J. et al. Large-scale silicon photonic circuits for optical phased arrays. IEEE Journal of Selected Topics in Quantum Electronics 20, 8201115 (2014).
    [147] Mahrous, H. et al. A compact 120 GHz monolithic silicon-on-silica electro-optic modulator. Optical and Quantum Electronics 52, 111 (2020). doi: 10.1007/s11082-020-2239-4
    [148] Jarrahi, M. et al. Optical switching based on high-speed phased array optical beam steering. Applied Physics Letters 92, 014106 (2008). doi: 10.1063/1.2831005
    [149] Porcel, M. A. G. et al. [INVITED] Silicon nitride photonic integration for visible light applications. Optics & Laser Technology 112, 299-306 (2019).
    [150] Su, T. H. et al. Experimental demonstration of interferometric imaging using photonic integrated circuits. Optics Express 25, 12653-12665 (2017). doi: 10.1364/OE.25.012653
    [151] Wang, H. J. et al. Broadband silicon nitride nanophotonic phased arrays for wide-angle beam steering. Optics Letters 46, 286-289 (2021). doi: 10.1364/OL.411820
    [152] Larocque, H. et al. Beam steering with ultracompact and low-power silicon resonator phase shifters. Optics Express 27, 34639-34654 (2019). doi: 10.1364/OE.27.034639
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures(6)

Research Summary

A review of holography for 3D display applications

The pioneers of holography, Gabor, Leith, Upatnieks, and Denisyuk, predicted very early that the ultimate 3D display will be based on this technique. This conviction was rooted on the fact that holography is the only approach that can render all optical cues interpreted by the human visual system. In this review article, Dr. Pierre-Alexandre Blanche from the University of Arizona, USA, is discussing the recent accomplishments made in the field of holographic 3D display. More specifically, the real time computation of holograms with neural networks and other algorithms, the efficient transmission and distribution of the information over long distance, and the new diffractive elements allowing the rendering of the holograms in   various 3D display setups.


show all

Article Metrics

Article views(8871) PDF downloads(1405) Citation(0) Citation counts are provided from Web of Science. The counts may vary by service, and are reliant on the availability of their data.

Holography, and the future of 3D display

  • Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA
  • Corresponding author:

    Pierre-Alexandre Blanche, pablanche@optics.arizona.edu

doi: https://doi.org/10.37188/lam.2021.028

Abstract: The pioneers of holography, Gabor, Leith, Upatnieks, and Denisyuk, predicted very early that the ultimate 3D display will be based on this technique. This conviction was rooted on the fact that holography is the only approach that can render all optical cues interpreted by the human visual system. Holographic 3D displays have been a dream chased after for many years, facing challenges on all fronts: computation, transmission, and rendering. With numbers such as 6.6 × 1015 flops required for calculations, 3 × 1015 b/s data rates, and 1.6 × 1012 phase pixels, the task has been daunting. This article is reviewing the recent accomplishments made in the field of holographic 3D display. Specifically, the new developments in machine learning and neural network algorithms demonstrating that computer-generated holograms approach real-time processing. A section also discuss the problem of data transmission that can arguably be solved using clever compression algorithms and optical fiber transmission lines. Finally, we introduce the last obstacle to holographic 3D display, which is is the rendering hardware. However, there is no further mystery. With larger and faster spatial light modulators (SLMs), holographic projection systems are constantly improving. The pixel count on liquid crystal on silicon (LCoS) as well as microelectromechanical systems (MEMS) phase displays is increasing by the millions, and new photonic integrated circuit phased arrays are achieving real progress. It is only a matter of time for these systems to leave the laboratory and enter the consumer world. The future of 3D displays is holographic, and it is happening now.

Research Summary

A review of holography for 3D display applications

The pioneers of holography, Gabor, Leith, Upatnieks, and Denisyuk, predicted very early that the ultimate 3D display will be based on this technique. This conviction was rooted on the fact that holography is the only approach that can render all optical cues interpreted by the human visual system. In this review article, Dr. Pierre-Alexandre Blanche from the University of Arizona, USA, is discussing the recent accomplishments made in the field of holographic 3D display. More specifically, the real time computation of holograms with neural networks and other algorithms, the efficient transmission and distribution of the information over long distance, and the new diffractive elements allowing the rendering of the holograms in   various 3D display setups.


show all
Introduction
  • Not long after the first demonstration of holographic images by Leith and Upatnieks1, as well as by Denisyuk2, a similar technique was used by De Bitetto and others to display motion picture holograms3, 4. As the name indicates, a motion picture hologram uses a rapid succession of static holographic images to reproduce the effect of movement in 3D. Because of these successes, it was hoped at the time that a holographic television would be developed soon after. Unfortunately, more than 50 years later, there is still no holographic television in sight.

    Compared to motion picture, a holographic display system needs to capture, transmit, and render 3D information. In the case of an interactive display system, such as a computer screen, there is the additional constraint that real-time manipulation of 3D data is required. This makes a universal holographic 3D display much more challenging to develop than the simple successive projection of pre-recorded holograms.

    To provide a sense of how difficult a holographic display is compared to other forms of telecommunication, it is useful to consider the commonly used data rate metric. Even though focusing on the data rate does not take into account some other technological aspects, such as rendering complexity, it does allow comparison over a large range of techniques. For a holographic display to have an acceptable field of view of perhaps $ \pm 45^\circ $ (diffraction angle $ \theta $), the law of diffraction indicates that the diffractive element size $ d $ for a central wavelength of $ \lambda = 500 $ nm should be on the order of 350 nm.

    $$ d = \frac{\lambda}{2 sin \theta} $$ (1)

    It should be noted that elements should be approximately 10 times smaller than predicted by equation 1 to achieve high-efficiency blazed diffraction gratings.

    Given a 70 cm diagonal-screen television set (i.e., $ \mathrm{50\; cm \;\times\; 50\; cm} $), the number of active elements must be on the order of $ 2\;\times\;10^{12} $. To accommodate human vision perception, the display requires a minimum refresh rate of 60 Hz (corresponding to the flicker vision threshold), and at least three colors to fill the eye chromaticity gamut. The gray-level resolution for a conventional display is generally at least 8 bits (256 levels). For the sake of comparison, we will use the same number of phase levels for a hologram, although phase levels are related to efficiency and not shades of gray in holograms. It is even possible to reconstruct a 3D image with a binary hologram by sacrificing some spatial bandwidth and efficiency5. The resulting data rate for such a display, excluding any type of compression algorithm, would be

    $$ \begin{split} & \mathrm{pxl\; nbr \times rep.\; rate \times res. \times colors}\\ & = 2\times10^{12}\times 60 \times 8 \times 3 \approx 3\times 10^{15}\; \mathrm{b/s.} \end{split} $$ (2)

    Fig. 1 plots the data rate in bits per second for different telecommunication systems according to the time of their introduction. Starting with the optical telegraph (or Chappe's semaphore) presented to Napoleon Bonaparte in 1798, the optical telegraph had a typical rate of transmission of approximately 2 to 3 symbols (196 different types) per minute, or 0.4 b/s. Consequently, the electrical telegraph, popularized in the early 1840s using Samuel Morse's code, achieved a rate of approximately 100 b/s. Graham Bell's telephone was introduced in 1876 and supported voice frequency transmission up to 64 kb/s6. The early NTSC black and white electronic television, available in the 1940s, had 525 interlaced lines and displayed images at a rate of 29.97 frames per second at a bit rate of 26 Mb/s7. The color NTSC format was introduced 10 years later and tripled the black and white bandwidth to accommodate red, green, and blue channels8. More recently, the digital video format makes it easier to establish the bit rate based on pixel count (excluding compression) with HDTV 720p@1.33 Gb/s in 1990, ultra-HDTV 2160p(4K)@12.7 Gb/s in 2010, and currently 4320p(8K)@47.8 Gb/s. Note that these values are for uncompressed data feeds, and for the sake of comparison, do not include any type of compression algorithm.

    Fig. 1  Stairway to holography: approximate bit rate magnitude of various telecommunication devices according to their year of introduction.

    The evolution of the bit rate for telecommunication devices plotted in Fig. 1 shows a trend that can be extrapolated to predict the emergence of holographic displays with a data rate of $ 3\;\times \;10^{15} $ b/s. This extrapolation estimates the emergence of a commercial holographic display by 2100. Although this extrapolation is indicative of the difficulties ahead, it is also very encouraging. The date of 2100 is by no means an inescapable natural law. Similar to the doubling of the transistor count on chips every year, this prediction can be affected in one way or another by the amount of effort invested in the research and development of holographic display technologies.

    In this manuscript, we investigate the reasons why holography is still perceived to be the ultimate technique to develop a commercial 3D display, review the progress that has been accomplished toward this goal, and discuss the missing technologies that are still needed to promote the emergence of such a 3D display.

    • Understanding the human visual system and how it perceives the third dimension is key to developing a 3D display911. The human visual system takes input from many different cues to determine depth perception. It should be noted that most of these cues originate from 2D phenomena. Among these are shading, shadowing, perspective, relative size, occlusion, blurriness, and haze. The example presented in Fig. 2 shows how three simple discs, presented on whatever 2D display you are reading this article on, are interpreted as 3D balls owing to these cues.

      Fig. 2  Examples of some of the 2D visual cues that affect depth perception.

      Because these 2D cues are processed by the human visual system to determine the depth of a scene, then a painting, a photograph, or a movie is intelligible as long as these cues are correctly reproduced. When they are not, this leads to optical illusions such as infinite staircases and other impossible shapes.

      The same applies to any 3D display system, which must, first and foremost, represent these 2D cues before introducing any additional cues. Additional 3D cues are stereo disparity, motion parallax, and accommodation. We briefly review these cues in the following sections.

    • Stereo disparity is the change in parallax of the scene observed between the left and right eyes. It only requires that two images be reproduced, and as such, is the most technologically manageable 3D cue. It is so manageable in fact that the introduction of stereoscopic displays pre-dates the invention of photography. The first system was invented by Sir Charles Wheatstone in the early 1830s using drawn images12. This was then followed by taking pictures from two positions, or with a camera having two objectives.

      When a stereo projection is meant for a single individual, such as a head-worn display, it is relatively easy to keep the left and right views separated. Images are separated by simply introducing a physical partition between both eyes13. For a larger audience, the separation between left and right views is often achieved by having the viewers use eyewear with different left and right lenses. The left and right image coding can be achieved using color (anaglyphs), orthogonal polarization, or alternating shutters14, 15.

      From a user perspective, the eyewear requirement for stereo display has been accepted in special venues such as theaters, where large productions continue to be released in stereoscopic 3D. However, the commercial failure of stereoscopic 3D television seems to indicate that for everyday experience, the public is not enthusiastic about wearing special glasses in their own living rooms16.

    • Autostereoscopic displays achieve stereoscopy without the need for special glasses. The left and right views are directly projected toward the viewer’s intended eyes using parallax barriers or a microlens array1719. To ensure that the correct eye intersects the correct projection, autostereoscopic systems require that the viewer be located at a particular position. This inconvenience has proven sufficient to limit the adoption of autostereoscopic 3D television by the consumer market20. It should also be noted that autostereoscopic systems with an eye tracking mechanism that mitigates the fixed viewer zones have been developed, but have not achieved wide popularity2123.

    • Motion parallax requires many views to be projected, allowing the viewer to see the correct parallax even when moving in front of the display. The density of the different views that are projected needs to be such that the autostereoscopic information is correctly reproduced. Therefore, at least two views per inter-pupillary distance are required. However, to achieve a smooth transition from one perspective to the next, a much larger density of views is required24. The optimum view density depends on the exact configuration of the display and the expected viewer distance, but numbers are on the order of one view per degree2527.

      In most of the literature, a display that reproduces motion parallax is called a "multiview" or "multi-view" display while a "light-field" display reconstructs 3D images based on the concept of ray-optics and integral imaging2832. In a multiview display, the display is designed such that the motion parallax can be reproduced smoothly when a viewer’s position changes. This is considered a multiview-type autostereoscopic display. However, when the display is also capable of reconstructing virtual or real images, it is usually called a light-field display.

      We can apply the same data rate computation introduced earlier for the different types of telecommunication devices (see Fig. 1) to a multiview (or light-field) display that reproduces motion parallax. In this case, we find that for a display with 2160p(4K) lateral resolution to reproduce motion parallax with a ±45° field of view, the bit rate is on the order of $ 12.7\; \times\; 90^2 = 10^{5}\; {\rm{Gb/s}} $. The square factor arises from the fact that both horizontal and vertical parallaxes are considered in this case.

      Because the human visual system involves a mostly horizontal inter-pupillary distance, and lateral movement is favored over vertical movement, horizontal parallax is more important than vertical parallax. The latter is often discarded in multiview displays to allow a lower data rate of $ 12.7 \;\times \;90 = 10^{3} \;{\rm{Gb/s}} $.

      When the viewer remains motionless in front of a multiview display, the observed parallax provides an experience similar to that of autostereoscopic displays33. However, because of the much larger number of views, a light-field display is not subject to the same limited number of view zones as autostereoscopic systems34. Therefore, user experience is much better, and acceptance is more likely.

      Considering their somewhat achievable data rate and advantages over auto-stereoscopy, multi-view and light-field displays are currently the subject of intense research3540. This technology certainly represents the next 3D display platform that will appear in the marketplace, and some specialized applications have already started to emerge41.

    • The vergence-accommodation conflict is the Achilles heel of all the display systems that we have introduced thus far: stereoscopic, autostereoscopic, multiview, and light-field (with some exception for the latter) and occurs when mismatched visual 3D cues are presented to an observer. The vergence-accommodation conflict occurs because the images projected by these displays are located at a fixed distance, thus producing a constant accommodation cue that cannot be adjusted, whereas vergence is provided by the parallax, which can be reproduced, and thus may vary within a scene. Disparity between accommodation and vergence cues can create a conflict in perception. This conflict leads to some visual discomfort, which is well documented in the literature4245.

      Light-field displays can reproduce some amount of accommodation when the ray density is sufficiently large. This condition is often referred to as a super-multiview46, 47. Accommodation occurs in a light-field display because the image plane can be moved in and out of the display plane. This is achieved by directing the light rays from different sections of the panel toward one voxel region, as shown in Fig. 3a.

      Fig. 3  Illustration of the projection of a voxel out of the emission plane by

      a a light-field display, and b a holographic display.

      However, there is some belief that if the view density keeps increasing in a light-field display, the accommodation distance can be extended at will. This belief arises from the extrapolation that light-field displays approximate a wavefront curvature by using line segments. If these segments are sufficiently small, they may become indistinguishable from the true wavefront curvature. Unfortunately, this ray-tracing simplification does not occur because diffraction along the pixel edges takes place, limiting the voxel resolution. Even with a pixel density in the 100s per degree, when an object is projected too far from the plane of the light-field display, it becomes blurry because of the diffraction among pixels. This diffraction effect cannot be avoided and intrinsically reduces the depth resolution and accommodation of light-field displays48, 49.

      To eliminate the diffraction phenomena experienced with smaller pixel sizes, strong coherence among pixels is required so that the light-field display becomes indistinguishable from holography.

      The difficulty of reproducing accommodation induces visual discomfort by having to limit the display depth of the field. To reproduce a voxel out of the plane of the display, the light should be focused at that point by the optical system. Without the capability to refocus subpixels at will, the light-field display can only produce a flat wavefront from the emission plane. As presented in Fig. 3a, when a light-field display attempts to reproduce a voxel that is too far away from the emission plane, the voxel invariably becomes blurry.

      To address this problem, researchers have developed multiplane light-field displays5052. This is possible because the plane of emission can be refocused by optical elements and moved along the view depth. However, this requires some multiplexing to generate different planes in time or space, which increases the bandwidth required by the system. Another aspect that should not be overlooked is that occlusions between different planes are difficult to control when there are many view zones53.

    • Volumetric displays have voxels located in 3D space and are affected by the same occlusion problem as a multi-plane light-field display. For both systems, the occlusions can only be correctly reproduced for one point of view54, 55. Some systems (both volumetric and light-field) use an eye tracking mechanism to re-calculate the occlusions and present the correct image wherever the viewer is located56. However, only one correct perspective can be achieved, precluding its application for multiple observers.

      In a volumetric display, the occlusion problem occurs because the emission of the voxel is omnidirectional, and there is no absorptive voxel. Nevertheless, volumetric displays have the advantage of being able to reproduce the field depth without resolution loss. They can be somewhat more natural to view when they do not use a screen to display an image. In this case, the image appears floating in thin air, which has a dramatic effect on the viewer’s perception55, 57, 58.

      Volumetric displays also have the disadvantage of not being capable of projecting images outside a limited volume. The image depth is bounded by that volume, and a deep landscape or object that seemingly reaches out of the display cannot be reproduced54.

      The mathematical computation of the bit rate for a volumetric display is as simple as multiplying the resolution of a 2D screen by the third dimension, refresh rate, and dynamic range. In Fig. 1, the data rate for a 4K volumetric display is

      $$ \begin{split} \begin{split} & \mathrm{x \times y \times z \times rep. rate \times res. \times colors }\\ & = 4096 \times 2160 \times 1000 \times 60 \times 8 \times 3 = 1.3 \times 10^{13}\; \mathrm{b/s.} \end{split} \end{split} $$ (3)

      However, because volumetric display setups are easily scalable, lower-resolution systems can be readily used to showcase the potential of the technology5961.

    Holographic 3D Display
    • Since the studies by Leith, Upatnieks, and Denisyuk, static holograms have demonstrated that the technique is capable of reproducing all the cues that the human visual system uses to comprehend the three dimensions1, 2. By using high quality photosensitive materials, it is now possible to copy existing artifacts and display convincing holographic reproductions in full color62, 63. The question that remains is how to do the same with a refreshable display.

      There are three fundamental problems to be solved to create a holographic television: the computation of the holographic pattern from the 3D information, the transmission of the data from where it is captured to where it needs to be displayed, and the reproduction of the holographic pattern on the screen to display the 3D image.

    • The calculation of the diffraction pattern from a 3D image is based on the physical model of light propagation. The Kirchhoff and Fresnel diffraction integrals provide the value of the field at a distance $ z $ from an aperture of arbitrary shape5. Early studies on computer generated holograms (CGHs) were undertaken in the late 1960s by Lohmann, Brown, and Lesem6466. Although the mathematical equations have been well-known for a very long time, their actual computation is far from trivial and has often been referred to as the "computational bottleneck"67. Even today, the number of operations per second (flops) needed to compute the size and location of the diffraction elements in real time is beyond the capabilities of any computer.

      To provide a sense of scale, and following the approach by Slinger et al.68, a brute force approach computation for a 720p ($ 1280\; \times \;720 $) holographic display requires $ 100 \;\times \;100 $ diffractive elements per pixel for full parallax, and 4000 multiplication and accumulation calculations per element at a rate of 60 Hz, and for three colors, this computes to $ 1280\; \times\; 720 \times 100\; \times \;100\; \times\; 4000\; \times \;3\; \times \;60 \;= $$ \; 6.6\; \mathrm{peta flops} $. Simplifications in the way that holograms are computed had to be made.

      The strongest simplification is to move away from the Fresnel integral and work in the far field with the Fraunhofer integral, which is a Fourier transform. By eliminating the $ z $ coordinate, the Fraunhofer integral yields the solution of the diffraction pattern for a given intensity field at infinity. Owing to the fast Fourier transform algorithm, a solution can be efficiently computed69. However, Fourier holograms have only a single image plane, so this simplification sacrifices image depth, which is not desirable for a 3D display.

      It should also be noted that the Fourier transform yields a real and an imaginary solution. These two components correspond to the amplitude and phase values of the hologram. Most of the time, the element used to display the diffractive pattern (such as a spatial light modulator) can only reproduce one or the other, but not both. This means that the result from a single Fourier transform will have a significant amount of noise when reproducing an image. Other sources of noise in holograms originate from the quantization error of the phase levels, diffraction in the pixel structure, and speckle caused by the random phase70.

      To reduce the noise and boost the signal, Gerchberg and Saxton developed an iterative algorithm (GS) in 197271. However, the GS algorithm only works for 2D input images and does not accept 3D information. Nevertheless, to obtain some image depth, it is possible to compute individual holograms for different discrete planes7274. This solution renders both vergence and accommodation. However, because the holograms for the different image planes have been computed separately from one another, and not as a whole 3D scene, occlusions can only be reproduced for a single perspective. This is the same problem previously mentioned regarding volumetric displays. More recently, some algorithms have been developed to address the occlusion problem in multi-plane hologram computations. Such algorithms have demonstrated the capability to render correct occlusions in a limited view zone7577.

      For 3D displays, hologram computations directly based on 3D models can be accomplished78. Although the detailed description of these algorithms is beyond the scope of this article, it should be noted that these algorithms can be separated into two broad categories: wavefront-based methods, and ray-based methods.

      In the ray-based technique, the hologram is calculated from incoherently captured 2D image planes of a 3D scene, and relies on the geometric optics formalism of light propagation. The ray-based methods include two distinct categories: holographic stereogram (HS) and multiple viewpoint projection (MVP)7981. Because they do not compute the wavefront propagation, HS and MVP techniques are much faster than wavefront-based methods and can render photorealistic images82. However, because the full wavefront of the object is not taken into account, ray-based methods can have difficulties rendering some of the 3D optical cues. Moreover, HS techniques experience some limitations in depth of field because the different views are combined incoherently48. Another drawback of the MVP approach is the need to capture or render a large number of images involving small increments in the location of the camera. Otherwise, the motion parallax can be jumpy, and occlusions are not represented well. In a sense, HS and MVP holograms are hybrids falling between light-field displays and holographic displays.

      In the wavefront-based methods, the propagation of the light wave is computed starting from a point light source that is illuminating either a point cloud or a polygon representation of the object. The CGH is calculated by simulating the interference between the wavefront coming from the object and another point light source or reference beam. The advantage of these methods is that they natively consider occlusion and parallax cues, so their rendering is accurate. However, this accuracy comes at the cost of high processing demand, as described previously48, 83, 76. This processing demand is driven by the fact that each point of the CGH must take into account the interference between each and every beam projected from the portion of the object that is visible from that location in the hologram plane. When the point on the hologram moves, the perspective of the object also moves correspondingly.

      Some of the computations for generating the CGH can be stored upfront in a look-up table84. This information can be retrieved much faster than if it is computed multiple times. There is a fine balance to achieve between large memory storage and computational complexity. To accomplish this balance, different types of algorithms have taken advantage of look-up tables at different stages of CGH computations8587.

      To handle the massive amount of data required to compute 3D holograms in a reasonable amount of time, computations are sometimes performed on specially built hardware accelerators88. This hardware is dedicated to the calculation of the Fresnel phase pattern86, 8891.

      Despite recent advances in the field of computer holography, it seems that from the published images available, the realism of the projected 3D images computed with wavefront-based algorithms requires significant improvement to become convincing (see Fig. 4)75, 92. This is by no means a criticism of the notable achievements accomplished in that discipline, it is rather a testament to how difficult it is to reproduce a fully detailed holographic image.

      Fig. 4  Examples of optical reconstructions of wavefront-based computer generated holograms from recently published articles.

      (1) Reproduced from [75]. Numerical and optical reconstruction results when focusing on the a, c head and b, d tail of the dragon. (2) From [93] presenting rendering images and optical reconstruction images of different surfaces a, d rough surface b, e smooth surface c, f rough surface with texture.

      In many cases, the holographic image computed using a wavefront-based method lacks texture (see Fig. 4(2)). This should not be surprising, considering that texture rendering considers the finest details of the material surface. This level of detail cannot yet be achieved by computer. A parallel can be drawn with the more familiar world of 2D animation, where early movies depicted blocky characters lacking luster but in modern times, the production team can use any level of photorealism that suits the artistic needs of the story (see for example,93).

      Techniques such as machine learning, neural networks, and artificial intelligence have recently been applied with great success to the computation of holograms9498. In the most general sense, the algorithms associated with these techniques work by training a computing network with some input images (ground truth) and use a camera to observe holograms generated by an optical system. The parameters of the code are optimized by the algorithm to minimize a loss function that forces the holograms to converge toward the original images. Once training is complete, the parameters are frozen, and the algorithm is eventually capable of calculating the hologram of any input image. These techniques are particularly effective at performing fast computations once the training period is complete and at solving the problems associated with texture rendering99, 96. However, for the most part, these algorithms work with 2D images, but are expected to soon be extended to 3D images.

    • The image captured for a holographic display can satisfy the minimum requirements of the human eye and does not have to use coherent illumination and resolve nanometric interference fringes, as is the case with static holograms.

      To suit the human eye accommodation, the 3D information to be reproduced can have a depth resolution of only a few centimeters, instead of the nanometers achieved with holography10. Such an image can even be compressed into a 3D mesh model overlaid by a texture pattern, as that used in modern video games. Video games handle this information together with the location of a virtual camera to display a 2D image. In the same way, the game engine could display a 3D image if the display required it. As a case in point, video games can be adjusted to and played using stereoscopic virtual reality headsets.

      We know that the amount of data to be transmitted to the display system is not overwhelming and can easily be accommodated by today's technology. However, from a 3D image, the computation of its hologram dramatically increases the amount of information because the diffraction pattern cannot be scaled to accommodate the lateral resolution of the human eye ($ \approx $ 1 arcminute or 0.3 mrad), but must instead be dimensioned according to the wavelength of visible light ($ \approx 500\; \mathrm{nm} $). This scaling can be appreciated in Fig. 1 by comparing the data rate of a volumetric display ($ \approx 10^{13} $ b/s) to a holographic display ($ \approx 10^{15} $ b/s)).

      Because of this increase in data size, it may be much more efficient to transmit the 3D image/model rather than the holographic pattern100, 101. In this case, the computation of the hologram should be performed at the client (receiver) location. This model is named the "thick client" because the computation is performed locally to avoid overwhelming the long-distance transmission medium. This means that the local site requires significant computational power to support this decoding (see the above section regarding Computed Generated Holograms).

      Alternatively, a remote bank of servers can be used to compute the hologram quickly and efficiently. In this configuration, only a "lean client" with limited processing power is needed at the receiver site, whose function is to process the local transmission for display purposes. Under this scenario, the near full amount of data ($ 10^{14} $ b/s) would need to be transmitted by the server to the client through a "fat pipe" network.

      A model of the transmission and reproduction of holographic images is presented in Fig. 5, along with the different orders of magnitude of the computation and data rate needed at each stage.

      Fig. 5  Schematic of a holographic television transmission process.

      Comparison between thick client and lean client architectures.

      We are already familiar with this type of lean client/fat pipe architecture as that used for cell phones and cloud computing. To support holography, the entire transmission network would have to be upgraded to support a $ 10^6 $ increase in data flow (from HDTV to holography in Fig. 1). This increase is not unreasonable considering that we have already experienced five cellphone network generations over the past 20 years102, 103.

      There is no clear advantage between the thick and lean client models. The main reason is that the need for the transmission of 3D holographic images and movies does not yet exist. However, it should be noted that the compression algorithm for hologram storage and transmission is not as effective as algorithms for natural images, such as JPEG and MPEG. This arises from the fact that the resolution of a diffraction pattern cannot be decreased without destroying the light interference it is suppose to generate and therefore the holographic image. Diffraction patterns need to be compressed using a near-lossless algorithm104107.

      Another important point regarding the transmission of holograms is that the computation of the interference pattern is specific to the display architecture. For the proper reproduction of a hologram, the computation of the interference pattern must consider whether the display is operating in full parallax versus horizontal parallax only, what are the exact illumination wavelengths, and what is the pixel density (among other parameters). Likewise, legacy displays that will be operating at the same time, such as 2D televisions, stereoscopic, auto-stereoscopic, and eventually volumetric displays, must be considered. To ensure compatibility among all devices, a lean client configuration would have to send the various display parameters to the server and receive the pre-calculated data in return. In the case of a thick client architecture, the server can invariably send the same model to the client, which is then further transformed locally. From this perspective, the thick client is simply another type of display that can be integrated into a lean client network, making these two concepts complementary rather than antagonistic.

    • In the context of an imaging device, the spatio-temporal product (STP) is the product of the number of pixels and the refresh rate. The introduction of the refresh rate in the equation is important because temporal multiplexing schemes can be applied to improve the display resolution (holographic or not). Multiplied by the pixel dynamic, the STP can be directly compared to the data rates used in Fig. 1. Given that we established that a holographic display should have a bit rate of $ 3\;\times\; 10^{15} $ b/s, the STP of the display device should have the same order of magnitude.

      It is technically feasible to create a large holographic display by tiling multiple spatial light modulators (SLMs). Using 4K SLMs that have a bit rate adapted for 2D imaging, i.e. 12.7 Gb/s, 230,000 SLMs would be required to reach $ 3\;\times \;10^{15} $ b/s, and 15,000 personal computers would be required to operate the number of screens involved.

      These ludicrous numbers again point to the difficulty of the task at hand. Nevertheless, researchers have demonstrated that such a tiling approach actually works, although on a smaller scale108111.

      Reduction of the STP can be achieved by reproducing the horizontal parallax only (HPO) and scanning the image vertically. HPO offers a reduction of the STP by a factor of approximately $ 10^3 $ compared to full parallax, and HPO displays do not need to maintain coherence between the different horizontal lines composing the 3D image. Because the human eye disparity is mostly horizontal, the loss of the vertical parallax in HPO holograms does not significantly impede 3D perception112. However, some other artifacts could be introduced if an HPO holographic display is used, such as astigmatism or the need for the viewer to stand at a given distance. Despite these issues, many researchers have taken advantage of the STP reduction offered by HPO to demonstrate holographic projection using a variety of systems113115.

      Another possible approach to reduce the STP of a full holographic system is to limit the eye box in which the hologram is projected. Using this technique, the light is directed toward the viewer using an eye-tracking system or a head-mounted display (AR/VR headset)116120. Knowing the location of the viewer dramatically reduces the calculation of the hologram because only a limited number of viewpoints need to be taken into account. Likewise, if the viewer is within a predefined region, the angular extent of the hologram (its diffraction angle) can be narrowed, and the number of diffractive pixels decreases. The advantage of this technique is that it does not sacrifice image quality or 3D cues.

    • There is no question that the introduction of the liquid crystal on silicon (LCoS) SLM has helped considerably in advancing holographic displays. LCoS SLMs have many convenient features for displaying diffraction patterns. With a pixel pitch down to a few microns, the diffraction angle can be as large as $ 10^\circ $ (see Eq. 1). SLMs can process large pixel counts, which helps to achieve high-resolution holograms. LCoS also has the advantage of being able to modulate the phase, which, along with an 8 bit phase-level resolution, achieves high diffraction efficiency. Unfortunately, the viscoelasticity of liquid crystals limits the refresh speed of LCoS to a few milliseconds. This speed is adequate for imaging purposes, even in color, but ultimately limits the STP of the LCoS SLM with respect to holographic applications121.

      To increase the STP of SLMs, it is possible to move away from the LCoS technology and use microelectromechanical systems (MEMS) instead. MEMS are composed of micro-mirrors that can be tilted or moved to interact with light. They can have the same number of pixels and approximately the same pixel pitch as LCoS. However, their refresh rate can be orders of magnitude higher122, 123. This increases their STP by the same factor, reducing the number of units needed to create a holographic display124126.

      Early MEMS examples include micro-ribbons developed by Sony that were used to construct a diffractive light modulator (or grating light valve). This technology boasted an impressive switching speed of 20 ns127, 128. However, micro-ribbons are one-dimensional, which requires another scanning mechanism to form a 2D image.

      At about the same time, Texas Instruments experimented with a phase modulator that moved pixels up and down to modulate the phase129. Unfortunately, this MEMS modulator was not commercialized. Instead, Texas Instruments invested in one of the most popular MEMS, the digital light processor (DLP)130.

      The DLP works by flipping its mirrors in one direction or the other to redirect light. This MEMS was developed for imaging purposes, such as projectors and televisions, and is very efficient in these configurations131. However, for holography, the DLP can only display amplitude holograms with maximum efficiency of 10%. Nevertheless, the DLP STP can reach an impressive 47.7 Gpixels/s with a resolution of $ 1920\;\times \;1080 $ pixels and a refresh rate of 23 kHz. Some chipsets have an even larger number of pixels supporting the 4K UHD resolution of $ 3840\;\times \;2160 $ pixels, but their refresh rate is slower (60 Hz), reducing the STP to 0.5 Gpixels/s.

      Most recently, Texas Instruments has rejuvenated its earlier attempt at a phase modulator and is developing a piston MEMS capable of achieving a much higher efficiency132134. This phase light modulator (PLM) should be extremely useful in the development of holographic 3D display systems. If the PLM is capable of operating at 20 kHz as some DLPs can, it will increase the STP of this MEMS by a factor of 100 compared to typical LCoS SLMs.

      Another approach that can increase the intensity of a hologram using low-efficiency devices is to use a refreshable holographic material. Refreshable materials, such as photorefractive polymers, can record the wavefront generated by the SLM and, owing to their high diffraction efficiency, amplify the intensity of the hologram. Video-rate holographic projection, as well as large holographic displays, have been demonstrated using this type of material34, 135, 136. However, it should be noted that these materials currently rely on an electronically addressable device (SLM, DLP, or other) to display the initial holographic pattern.

      Considering that the STP is the key to unlocking a practical holographic 3D display, the approach taken by a group of researchers at MIT and BYU (initiated before the DLP was available) was to start with the device that had the largest STP at the time, acousto-optic material (AOM)137, 138. Regarding acousto-optic materials, the propagation of a sound wave creates a density modulation that diffracts light. If the sound wave is correctly programmed, the diffracted light can form a holographic image. In its waveguide format, the acousto-optic modulator allows for a longer interaction length between light and the acoustic wave generated, which further increases the STP139141. A single leaky acousto-optic waveguide can have a 50 MHz usable bandwidth per color, which corresponds to 1.67 Mpixels at 30 Hz. By fabricating multiple waveguide channels in a single crystal, these numbers can easily be increased by several thousands to reach an STP of 50 Gpixels/s. Although the initial demonstrations using AOM provided horizontal parallax only, it is theoretically feasible to feed the different waveguides using a single laser source and control the phase such that horizontal and vertical coherent beam steering can be achieved142, 143.

      Another high-STP device is the phased array photonic integrated circuit (PIC)144, 145. In this approach, nanophotonic phased arrays are built by recording branching waveguides on a photonic wafer (see Fig. 6). The waveguides are organized such that they distribute light projected from a single source over a 2-dimensional grid. The phase at the end of each waveguide can be adjusted by an electro-optic or thermo-optic phase regulator. The light is extracted orthogonally from the wafer by a grating outcoupler terminating each waveguide. Analogous to phased array radar, the grating outcoupler is also called an optical antenna.

      Fig. 6  Schematic of a photonic integrated circuit optical phased array. A single coherent laser source is directed inside a waveguide, from which light is extracted by multiple grating couplers (acting as light antennae). The phase at each antenna can be tuned using a phase modulator to create a hologram.

      The advantage of PIC phased array technology is the very high frequency at which the phase can be modulated. Electro-optic modulators can reach one hundred GHz146, 147. This inherently elevates the data rate to the $ 10^{10} $ b/s order. Using an array with $ 300\;\times \;300 $ antennae would achieve the $ 10^{15} $ b/s required for a holographic display. As we will discuss, the present difficulties of photonic phased arrays are the wafer material, the gaps between antennae, and the phase accuracy between antennae.

      The preferred material for PIC is silicon, which does not transmit visible light. Other materials with better transmission in the visible wavelengths should be used for display purposes. Silicon nitride or silica platforms have already been explored for optical phased arrays in the literature, but remain in their experimental phase145, 146, 148, 149.

      Compared to MEMS and LCoS that have a fill factor above 90%, the fill factor of the phased array is fairly low, at approximately 25%. The fill factor affects the diffraction efficiency owing to the presence of side-lobe emissions that cannot be canceled if the antennae are too far apart. This separation is due to the limited turn radius of the waveguide and the required separation between waveguide elements to avoid cross-coupling150. Both factors, turn radius and waveguide separation, are dictated by the difference in the index of refraction between the inside and outside of the waveguide. A larger index difference would allow for a larger fill factor.

      The phase control of the pixels is better in LCoS than in both MEMS and phased arrays. The LCoS phase is analog and proportional to the applied voltage and is therefore uniform across pixels. In contrast, current MEMS micro-mirror levels are discrete and limited to 4 bits, and exhibit some nonlinearity134. For phased arrays, the phase control is analog and accurate, but has to be characterized for each element individually owing to manufacturing inconsistencies145.

      In summary, none of the current SLM technologies is sufficiently mature to meet all the criteria required for a large-size, high-definition holographic 3D display. This should not overshadow the considerable progress that has been made over the past years, bringing the end goal ever closer.

    Conclusion
    • Holography is still considered as the ultimate technology that will enable rendering of all the optical cues needed for the human visual system to see projected images in 3D. All other technologies, such as (auto) stereoscopy, light-field, or volumetric displays suffer from trade-offs that limit 3D rendering. Nonetheless, these technologies will likely prove to be stepping stones leading to better visual comfort until holographic displays are achieved.

      Some of the doors that were preventing holographic television from being made possible only a few years ago have already been unlocked. The fast computation of 3D holograms to properly control occlusions and parallax is now within reach as is a solution to the problem of data transmission. The exact architecture of the network (thick or lean client) is unclear, but higher compression rates and ever faster telecommunication infrastructures supporting the Internet mobile communications make streaming the data for a holographic television feasible, if not yet accessible.

      However, some challenges remain to be solved. The two main obstacles at the time this manuscript was written are the computation of photorealistic 3D holograms in a reasonable amount of time, and a suitable electronic device for the reproduction of large holographic 3D images with high resolution.

      For the pioneers of holography, Gabor, Leith, Upatnieks, and Denisyuk, the challenge of controlling diffractive pixels could only have been a physicist’s dream. This dream has now been transformed into an engineering nightmare, as devices that can project holographic images exist, but scaling the format to project large images can be surprisingly difficult. Computational time becomes prohibitive, and controlling trillions of pixels currently requires an extremely large number of graphic cards.

      Nonetheless, the difficulty of projecting large holograms will soon no longer be in the engineering realm, but will rather transform into an economic problem. Initially, the hardware needed to build a holographic television will be too expensive to be successfully commercialized as a television set. Once price becomes affordable, the holographic television will face the same hurdle that each new media is bumping into, which is the availability of correctly formatted content. There is no advantage in owning a holographic television if creators are still producing 2D movies exclusively. For these reasons, it is likely that the market will move incrementally, starting with HPO multiview, expanding to light-field and full parallax, and finally reaching holography.

      To achieve this dream of the perfect display, we should remember that the bit rate progression reported in Fig. 1 is not an immovable fact of nature. It is a testament to human ingenuity and hard work. This exponential growth can be influenced in one way or another by our own actions. Ultimately, where there is a will, there is a way, and the desire for a truly immersive visual experience is ingrained in human nature. It is this desire that will make the holographic television a reality sooner than later.

    Conflict of interest
    • There are no conflicts to declare.

Reference (152)

Catalog

    /

    DownLoad:  Full-Size Img PowerPoint
    Return
    Return