Gradient-based Optimization of Time-Multiplexed Binary Computer-Generated Holograms by Digital Mirror Device - Digital Holography and Three-Dimensional Imaging at OSA Imaging and Applied Optics Congress (oral presentation by Kenta Yamamoto)

This slide was presented in "Computer Generated Holograms III (DTh7C)" at t OSA Imaging and Applied Optics Congress.
https://www.osa.org/en-us/meetings/osa_meetings/osa_imaging_and_applied_optics_congress/

【Publication】
Yamamoto, Kenta and Ochiai, Yoichi. “Gradient-based Optimization of Time-Multiplexed Binary Computer-Generated Holograms by Digital Mirror Device.” Digital Holography and Three-Dimensional Imaging. Optical Society of America, 2021. (to appear)
https://digitalnature.slis.tsukuba.ac.jp/2021/07/multiple-binary-hologram-optimization-publication/

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2021/07/multiple-binary-hologram-optimization/

【Presenter】
Kenta Yamamoto (山本健太)
University of Tsukuba
Graduate School of Human Comprehensive Science.
Digital Nature Group (Yoichi Ochiai)

【Abstract】
We propose a gradient-based optimization method for time-multiplexed binary holograms displayed on a digital mirror device. Optimized binary holograms can be used to reconstruct high-quality images.

Transcript

1. Gradient-based Optimization of Time-Multiplexed Binary Computer-Generated Holograms by Digital Mirror Device Kenta Yamamoto1, Yoichi Ochiai1 1University of Tsukuba, Digital Nature Group
2. 2 Overview of Our Work After Aberration Correction Process summary diagram of the proposed method. "Gradient-based Optimization Method of Time-Multiplexed
 Binary Computer-Generated Holograms".
3. Background
4. 4 Background After Aberration Correction For Displaying 3D Images, Computer-Generated Holography (CGH) is important. Holographic display, holographic projector, holographic near-eye display are based on CGH. Holographic Display [Yaras 2010] Holographic Projector [Buckley 2011] Holographic Near-Eye Display [Maimone 2017]
5. 5 Background After Aberration Correction For full-colorization, high refresh rate equipment is essential. Phase SLMs are applied for current holographic displays due to the diffraction efficiency. However, the refresh rate is still slow (regularly up to 60Hz). LCOS SLM High Diffraction Efficiency Low Refresh Rate Digital Mirror Device Low Diffraction Efficiency High Refresh Rate
6. 6 Background After Aberration Correction Therefore, DMDs have the potential for full-color holographic display. However, the basic problem is how to obtain a high-definition reproduced image with binary holograms because DMD can only display in binary. LCOS SLM High Diffraction Efficiency Low Refresh Rate Digital Mirror Device Low Diffraction Efficiency High Refresh Rate
7. 7 Background After Aberration Correction Hologram Binarization Method 1. Sign Thresholding 2. Error Diffusion 3. Direct Binary Search 4. Iterative Method 5. Down Sampling 6. Advanced Down Sampling and Iterative Method
8. 8 Background After Aberration Correction Sign Thresholding This is a very simple technique. If the real part of the original hologram Hc is 0 or more, it is 1, otherwise it is 0. Hc(m,n): originam hologram Hb(m,n): binarized hologram
9. 9 Background After Aberration Correction Error Diffusion A method of sharing the error with surrounding pixels. <simple algorithm> 1. E0 = 0 2. H'0 = H0 - E0 (repeat 3.-5.) 3. Hout0 = binarize(H'0) 4. E = Hout0 - H0 5. H'1 = H1 - E ( binarize(): ex. Sign Thresholding ) Reiner Eschbach. "Comparison of error diffusion methods for computer-generated holograms". 1991.
10. 10 Background After Aberration Correction Direct Binary Search A method of updating a binary hologram according to the change in MSE when each pixel is inverted. After verifying all pixels, the process ends. Seldowitz, et al. "Synthesis of digital holograms by direct binary search". 1987.
11. 11 Background After Aberration Correction Piestun et al. "On-axis binary-amplitude computer generated holograms". 1997. Iterative Method A method of reaching the optimum hologram while repeating propagation by applying restrictions before and after propagation.
12. 12 Background After Aberration Correction Down Sampling A method of thresholding after down-sampling the target image. In "Computer generation of binary Fresnel holography", downsampling is performed periodically. Tsang et al. "Computer generation of binary Fresnel holography". 2011.
13. 13 Background After Aberration Correction Localized Random Down-Sampling and Adaptive Intensity Accumulation For downsampling, a method of randomly selecting pixels is adopted. Better hologram optimization is achieved by generating an adaptive mask according to the accumulated intensity. Liu et al. "3D display by binary computer-generated holograms with localized random down-sampling and adaptive intensity accumulation". 2020.
14. 14 Background After Aberration Correction Gradient-based hologram optimization In recent years, gradient-based hologram optimization has achieved high accuracy. "3D computer-generated holography by non-convex optimization" (2017) "High resolution étendue expansion for holographic displays" (2020) "Wirtinger holography for near-eye displays" (2019)
15. 15 Background After Aberration Correction Methods using automatic differentiation have incredible high accuracy algorithms. For example: "Neural Holography" and "Diff-PAT". Peng et al. "Neural Holography with Camera-in-the-loop Training". 2020. Fushimi and Yamamoto et al. "Acoustic hologram optimisation using automatic differentiation". 2021. Input: φ Differentiable Propagation Calculation: ̂ f Output: ̂ f(ϕ) Neural Holography Diff-PAT
16. 16 Background After Aberration Correction We aim to realize a high-definition reproduced image with a small number of holograms 
 by combining time-multiplexed binary holograms and gradient-based optimization using automatic differentiation. Time-Multiplexed Binary Holograms Gradient-based Optimization using Automatic Differentiation
17. Method
18. 18 Method After Aberration Correction Hologram optimization method using automatic differentiation. 1. Propagation Calculation from Initial Random Hologram 2. Calculate Difference (Loss) between Propagated Image and Target Image 3. Gradient for each pixel is derived 4. Repeat
19. 19 Method After Aberration Correction Impossibility of gradient derivation in binary hologram. When the gradient of the binarize step function is derived, it becomes 0 (at t=0, ∞).
20. 20 Method After Aberration Correction Tensorflow was used for the automatic differentiation package
 (because it supports 2D Fourier Transform). Tensorflow has a function called "custom gradient", which allows to specify the gradient. When the gradient of the binarize function is fixed to 1, the optimization came to converge.
21. 21 Method After Aberration Correction This is the process for time-multiplexed.
22. Results
23. 23 Simulation After Aberration Correction Simulation Results (c) original image (d) N=2, PSNR: 22.55 (e) N=3, PSNR: 25.75 (f) N=5 , PSNR: 28.85 (g) N=10, PSNR: 29.25
24. 24 Simulation After Aberration Correction Results for each iteration of Loss and PSNR. (a) loss and iteration (b) PSNR and iteration N=10 N=5 N=3 N=2 N=10 N=5 N=3 N=2
25. 25 Experiment After Aberration Correction Real-environment experiment to confirm optimized hologram. Texas Instruments DLP 9000 was used for DMD. (g) N=10, PSNR: 29.25 (h) captured image (N=10) (c) original image
26. Additional Experiment
27. 27 Additional Experiment After Aberration Correction When preparing this presentation, we noticed that a paper about "Differentiable Binarization" was published in 2020 in AAAI. Pi,j: Input, Ti,j: Threshold, Bi,j: Binary Map, k: 50 (empirically derived) DB: Differentiable Binarization SB: Standard Binarization Liao et al. "Real-time Scene Text Detection with Differentiable Binarization". 2020.
28. 28 Additional Experiment After Aberration Correction Simulation results optimized using this function. Even if this function is used, the binzary hologram is optimized, but the result is better when the custom gradient is used. PSNR: 21.95 Five optimized binary holograms.
29. Summary and Future Work
30. 30 Summary After Aberration Correction Here is the summary of our work. 1. A gradient-based optimization method using automatic differentiation was applied to the derivation of time-multiplexed binary holograms. 2. We conducted simulations and real-environment experiments and showed that the optimization results are useful. (g) N=10, PSNR: 29.25 (h) captured image (N=10) (c) original image
31. 31 Future Work After Aberration Correction The following three issues should be addressed in the future. 1. Because we have not compared it with the existing methods, we will compare it and verify whether it has an advantage. 2. We will investigate the cause of poor results when using the differentiable binary function. 3. We will apply to optimization of full-color holograms and build optical systems.
32. Gradient-based Optimization of Time-Multiplexed Binary Computer-Generated Holograms by Digital Mirror Device Kenta Yamamoto1, Yoichi Ochiai1 1University of Tsukuba, Digital Nature Group