SSII2025 [SS1] レンズレスカメラ

1. Lensless cameras Tomoya Nakamura, The University of Osaka
2. Lensless imaging [Opt.Exp. 2020, IEEE TCI 2023, Appl.Opt. 2024] 3D microscopy [Opt. Lett. 2018, Biomed. Opt. Exp. 2022 w/ Prof. Yuichi Kozawa] Holography [Opt. Lett. 2016, Opt. Exp. 2018] Photopolymer [Opt. Exp. 2018, ITE MTA 2021] Display Compressive sensing [Sensors 2019, Opt. Contin. 2023, Opt. Rev. 2024] Gigapixel camera [Appl. Opt. 2013 w/ Prof. David Brady] [mm] Monocentric objective lens Megapixel micro-cameras Assoc. Prof. at XR Group at UOsaka, Computational Imaging & Holography CGH z to x CGH fabricated as a amplitude mask Transparent off-axis mirror fabricated using photopolymer DOE ToF cam PSF w/ subpixel shift SR depth map
3. Computational imaging 3 Extending the freedom of design in imaging system involving computation
4. Lensless camera 4 Camera Lensless camera To reconstruct image stably, coded optical element is inserted near the sensor H. Kawachi, T. Nakamura, J. Neto, Y. Makihara, Y. Yagi, “Revolutionizing Photography: Demonstration of Lensless Imaging by Replacing the Lenses with a Thin Radial Coded Mask in Consumer-Grade Cameras,” in IEEE ICASSP [Show & Tell Demos], DEMO-2A.1, 2024.
5. Lensless camera 5 Captured Code information (PSF)
6. Lensless camera 6 Captured Code information (PSF) k = 2 k = 4 k = 16 k = 30 k = 8 k = 1 Iterative reconstruction algorithm ̂ f = argmin g − h * f 2 2 + τΦ( f) f
7. H. Kawachi, T. Nakamura, J. Neto, Y. Makihara, Y. Yagi, “Revolutionizing Photography: Demonstration of Lensless Imaging by Replacing the Lenses with a Thin Radial Coded Mask in Consumer-Grade Cameras,” 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2024) [Show & Tell Demos], Seoul, No. DEMO-2A.1, Apr. 2024. Demonstration@IEEE ICASSP
8. 8 Computational image reconstruction algorithms Wiener filter Machine learning CPU CPU CPU filtering Measurement Measurement Reconstruction Measurement Reconstruction CPU CPU CPU -1 Pretrained network Error minimization with handcrafted prior Error minimization with deep image prior Regularize Error Update Update Estimate Estimate Measurement Error Estimate Estimate Generative CNN e.g., ADMM e.g., Deep Image Prior (DIP) தଜ༑࠸, “ޫූ߸Խͱ৘ใ࠶ߏ੒Λ༥߹ͨ͠ϨϯζϨεΧϝϥٕज़,” ೔ຊόʔνϟϧϦΞϦςΟֶձࢽ, Vol. 29, No. 3, pp. 20-26, (2024).
9. Experimental comparison of algorithm 9 Measurement data ADMM with TV prior (Classical compressive sensing) Generative prior Reconstruction with generative prior visually outperforms classical one, though it takes 48x longer time! PSF 40 sec. 32 min. Coded aperture by glass plate with chromium deposition
10. Extended depth-of- fi eld lensless imaging José Neto, Tomoya Nakamura*, Yasushi Makihara, and Yasushi Yagi, “Extended Depth-of-Field Lensless Imaging using an Optimized Radial Mask,” IEEE Transactions on Computational Imaging, Vol. 9, pp. 857-868 (2023).
11. PSF 11 Optical model of coded imaging • Lateral space: 2D Convolution • OTF determines the performance • Axial space: scaling and summation • Depth variance of PSF determines the depth-of- fi eld g = Crop [∑ z hz * fz] g = Crop[h * f] Captured PSF Object Coded optics Image sensor
12. 12 Depth invariant PSF by lensless optics PSF (Optical shadow of a coded mask) Image sensor
13. 13 Depth invariant PSF by lensless optics PSF (Optical shadow of a coded mask) Image sensor
14. 14 Depth invariant PSF by lensless optics PSF (Optical shadow of a coded mask) Image sensor
15. 15 Extended depth-of- fi eld lensless imaging Reconstruction algorithm Sensor measurement Objects placed at multiple depths Limited DOF reconstructed image Random coded mask Image sensor Extended DOF reconstructed image Optimized radial coded mask A B A a) Optical axis
16. 16 PSF optimization • PSF pattern along rotational direction was optimized for maximizing MTF • Binary and randomized pattern was obtained
17. • Reconstruct lensless image using rear-plane PSF -> Our method successfully reconstructs front object at once • Can be applied to close-up imaging with thin optical hardware Experiment
18. Multi-layering of coded mask in lensless cameras Tomoya Nakamura*, Reina Kato, Kazuya Iwata, Yasushi Makihara, and Yasushi Yagi, “Multi-layer lensless camera for improving the condition number,” Applied Optics, Vol. 28, pp. G9-G17 (2024).
19. Multi-layering of coded aperture in lensless camera • Existing lensless imaging systems employ optical design with a single coded aperture (CA) Lensless imaging CPU t x = t Coded image Reconstructed image Coded image Reconstructed image Coded aperture Stacked coded apertures Coded image Reconstructed image Multi-layer lensless imaging CPU x = Inverse problem Inverse problem • Technically, of course there is a room for increasing the number of the CA. Is there any bene fi ts to do so?
20. Possible bene fi ts (improvement of condition number) Coded aperture Lensless camera Multi-layer lensless camera Light Sensor • Let’s think about inserting additional CAs inside a single CA system. • Propagation distance of shadow can be decreased. • So there is a possibility to improve the imaging performance of the lensless camera
 - optically, contributes to fi ner impulse response
 - mathematically, contributes to better system matrix with smaller condition number • This should include possibility, but not so simple because it also affects the open ratio of CAs to keep the light ef fi ciency. Investigation is needed.
21. Simulation: observation of system matrix by ray tracing       Single CA σ = 0.0 Close up of submatrix Output dim. Input dim. σ = 0.5 σ = 1.0 σ = 1.5 2-layer CAs 4-layer CAs  max max max max • Prior to quantitative analysis, we observed the system matrix by ray tracing considering diffraction Blur amount of shadow
 (Total thickness of optics) Total counts of CA • Not shift invariant • Includes fi ner structure
22. Simulation: singular values and condition numbers in matrix Singular value Index Single CA 2-layer CAs 4-layer CAs σ = 1.0 σ = 1.5 Improvement Improvement Improvement σ = 0.0 Singular value Index Single CA 2-layer CAs 4-layer CAs σ = 0.5 σ = 1.0 σ = 1.5 • By simulations, we calculated the singular values of the matrix and corresponding condition numbers • We con fi rmed they were improved even when we fi xed the light ef fi ciency and total thickness of the lensless camera
23. Simulation: image reconstruction with noise 
    E# 
    E# 
    E# 
    E# 
    E# 
    E# Ground truth Captured data Reconstructed data σ = 1.0 max 0 σ = 1.5 σ = 1.0 σ = 1.5 Single CA 2-layer CAs 4-layer CAs Ground truth Captured data Reconstructed data σ = 1.0 max 0 σ = 1.5 σ = 1.0 σ = 1.5 Single CA • Sensing was simulated by ray tracing considering diffraction with 40dB noise • PSNR in reconstruction images was improved between 2-3 dB • We also con fi rmed condition number in the system matrix was also improved (see paper for detail) Reconstructed by TwIST algorithm, a sort of gradient method including regularization technique
24. Experimental setup Top view Front view Coded aperture Heat sink Plate holder Sensor Experimental setup Multi-layer lensless camera Monitor • We also prove this improvement by optical experiments using a prototype with 2-layers CAs CA was implemented by laser lithography with chromium deposition on a glass plate
 (~1um resolution)
25. Experiment: condition number and imaging quality Ground truth Captured Reconstructed max 0 Single CA 2-layer CAs Binarized Ground truth Captured Reconstructed max 0 ngle CA Binarized • Condition number was improved from 43.6 × 105 to 8.88 × 105 • Since this is the optical experiment, we didn’t have ground-truth and cannot calculate PSNR.
 Instead, we calculated the matching rate of the input and reconstructed binary data.
 The rate was improved from 0.77 to 0.81.
26. Conclusions, Q&A Radial mask enables extended depth-of- fi eld in lensless imaging Multilayering of coded mask improves quality of lensless imaging