CVIM_2022_01

Computational imaging based on multi-tap CMOS image sensors and multi-aperture optics

Transcript

1. Computational imaging based on multi- tap CMOS image sensors and multi- aperture optics Keiichiro Kagawa Research Institute of Electronics, Shizuoka University
2.  B.Eng. & M.Eng. degrees from Osaka University, Japan, in 1996 and 1998, respectively  Ph.D. degree from Osaka University, Japan, in 2001  Was with Nara Institute of Science and Technology, Japan, 2001-2007  Was with Osaka University, Japan, 2007-2011  Have been with Shizuoka University, Japan, since 2011  My interests are in applications and development of CMOS image sensors for biomedical imaging and computational imaging  Email: kagawa@idl.rie.shizuoka.ac.jp Self Introduction
3. Outline  Low-level processing in the charge domain ◼ Compressive sensing ◼ Multi-tap CMOS pixels  Ultra-high-speed filming ◼ Single-shot ◼ Multi-exposure for time-of-flight range imaging
4. Contemporary CMOS image sensor CMOS VLSI analog/digital circuits CCD’s legacy pinned photodiode + High performance Gain amplifier ADC High frame rate High pixel count High photosensitivity Low noise Low dark current
5. How to Implement Functions in Camera Systems Ordinary pixel array ＋ Rich digital signal processing circuits Pixel array with in-pixel rich processing circuits Low-level focal-plane processing without in- pixel circuits Simple and small pixel Complicated and large pixel Post-processing Reduced data Results Results Results Conventional Charge-domain signal processing Smart pixel
6.  Position: x, y, z  Direction: kx, ky, kz  Time: t  Wavelength: λ  Polarization: S0 , S1 , S2 , S3 (Stokes parameters)  … Compression of Optical Signals
7. Compressive Sensing (CS) More sampling points are reproduced from less samplings a Observation matrix Measured signal Original signal b (K-sparse)
8. Mathematical Formulation a b (K-sparse) Observation matrix Measured signal Original signal min𝑥 = ෍ 𝑖 𝐷𝑖 𝒙 1 1, s. t. 𝐴𝒙 = 𝒚 min𝑥 = ෍ 𝑖 𝐷𝑖 𝒙 1 1 + 1 2 𝐴𝒙 − 𝒚 2 2, s. t. 𝐴𝒙 = 𝒚 Total variation (TV) is used
9. Fundamental Operation: Inner Product Operation a Observation matrix Measured signal Original signal 𝑦𝑖 = 𝒂𝑖 ∙ 𝒙 Inner product (or correlation) b (K-sparse)
10.  Single-pixel imaging Single detector  Compressive video Narrower bandwidth  Hyper-spectrum camera A lot of wavelength spectral bands  Lensless camera Ultra-thin  X-ray CT Lower X-ray exposure or shorter acquisition time  … Examples of CS applications
11. Charge Handling in CMOS Image Sensor Pixels Accumulate Transfer Sort Drain Read Basic operations Optional
12. 4T active pixel  Pinned photodiode (Buried and fully depleted)  Complete charge transfer  Floating diffusion amplifier  Low dark current  Noiseless charge handling  High conversion gain p n n + Depletion region p + SiO2
13. Charge Modulators Sort Drain + = Used in time-of-flight depth image sensors and computational image sensors Temporal shutter a Time x a
14. Inner Product Operation by Charge Modulators Sort Drain + = Temporal shutter a Time x a 𝑦𝑖 = 𝒂𝑖 ∙ 𝒙 Temporal inner product (or correlation)
15.  Noiseless operation ◼ By complete charge transfer with pinned photo diodes  No processing circuit is necessary ◼ Low-level signal processing is done in the charge domain Benefits of Charge Modulators
16. Related Work: Compressive Video [Hitomi et al. (2011)] [Sonoda et al. (2016)] Add the address line +High sensitivity High resolution - Temporal constraint Share the control line +High sensitivity High resolution -Spatial constraint [Luo et al. (2019)] In-pixel shutter memory +No constraint -Lower sensitivity Lower resolution
17. Related Work: Spatial Compression (in Voltage) [Oike et al. (2012)]
18. Ultra-High-Speed Image Sensors Crack propagation Shock-wave propagation Plasma emission Crash test Continuous readout Burst readout Frame rate 1kfps 10kfps 10Mfps 100Mfps 1Gfps 20Mfps 16.7Mfps For example 3.5kfps High-speed ADC Column memory CMOS In-pixel memory CCD In-pixel memory CMOS Our method: in-pixel CS CMOS 303Mfps 125Mfps
19. Bottleneck: In-Pixel Memory CCD Multi-stage charge transfer - High gate voltages for high transfer efficiency - High power consumption Bottleneck
20. Bottleneck: Column Memory CMOS Long shared readout line - Parasitic resistance and capacitance Bottleneck High current density - Electromigration
21.  DO NOT use analog amplification circuits  To reduce the number of memory elements operated at the same time Key to Realize Ultra-Fast Frame Rates
22. Our Work: Multi-Aperture and Macro-Pixel CS Multi-aperture Macro pixel Shutter pattern Per aperture Per subpixel Disparity Yes No Lens Special lens array Ordinary lenses Macro pixel Subpixels Apertures
23. Architecture Comparison Pixel array Multi-frame buffer Conventional Output Readout PD CDS Memory Pixel Focal-plane shutter controller Our multi-aperture CS Pixel array Pixel array Pixel array Output Readout Sub image sensors Post-processing
24. Comparison of Time-Window Functions Multi-aperture framing A1 A2 A3 A4 Observation period Multi-aperture compressive A1 A2 A3 A4 Observation period Conventional burst readout Observation period F1 F2 F3 F4 Time Efficient sampling
25.  Only the speed of one-step charge transfer determines the maximum frame rate.  More frames or extended depth range is obtained in single-shot filming and time-of-flight depth imaging, respectively. Benefits of CS in Ultra-High-Speed Filming
26. Compressive Sensing (CS) More sampling points are reproduced from less samplings. a Observation matrix Measured signal Original signal b (K-sparse)
27. Multi-Aperture CS a apertures b frames (sparse with TV) = x A y Observation matrix Measured signal Original signal Time Time
28. Comparison Pixel array Multi-frame buffer Noise reduction & Signal transfer causes a pause ↓ Limited frame rate Conventional Single-aperture Output Readout Multi-aperture CS Solving an inverse problem Focal plane time-coded multiple exposure Multi-aperture Pixel array Pixel array Pixel array Output Pixel array itself works as a frame buffer No pause Higher frame rate Noise reduction & readout
29. Considerations on pixel design  No barriers and dips  Fringing electric field p n n + SiO 2 n+ Fringing electric field Depth p n n + Depletion region p + SiO 2 Dip Barrier Diffusion Drift
30. Lateral Electric Field Charge Modulator (LEFM) Conventional CMOS LEFM
31. Multi-Tap to Increase # of Measurements Multi-tap a 1 a 2 a n x
32. Examples of Multi-tap LEFMs  Multiple charge storage diodes per pixel  Low noise by true CDS  High-speed charge modulation
33. 200Mfps Multi-Aperture CS Camera SPI Vertical scanner LVDS AP controller PLL Clock tree Bias circuit OB 5×3 APs Read circuit 5x3 lens array
34. Demonstration of single-shot UHS video capturing Focused laser beam Air break down plasma Camera APD Stop trigger FPGA CAPTURE_START • 15 apertures • 32-bit shutter pattern • 200Mfps capturing • Capturing is stopped by “Stop trigger”
35. Experimental system APD Camera Plasma Laser Objective lens 2mm 5×3 apertures
36. Results (Single shot) Compressed 15 images Time resolved 32 images (comp. ratio ~ 47%) Chengbo Li, http://www.caam.rice.edu/~optimization/L1/TVAL3/ #18 Frame #12 5ns Time Applied shutter patterns Solving inverse problem by TVAL3 Ideal pattern Impulse response Minimum pulse width = Time resolution
37. Our Macro-Pixel CS camera: 303Mfps Object Time Macropixel Convolution with spatial PSF Temporally compressed images Reproduced images PD 1 2 3 4 PD 1 2 3 4 PD 1 2 3 4 PD 1 2 3 4 * Temporal coded shutter Subpixel (SP)-1 SP-3 SP-2 SP-4 Tap Single-aperture imaging lens Temporally compressive CMOS image sensor Time
38. Example of Temporal Coded Shutter Time Tap SP 1 2 3 4 Bit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 Macropixel PD 1 2 3 4 PD 1 2 3 4 PD 1 2 3 4 PD 1 2 3 4 Tap
39. Simulation of compression and reproduction 32 original images Time Captured compressed images @compression ratio of 800% 32 reproduced images @PSNR=34.1dB
40. PSF and shutters ISSCC 2022 - Forum X.Y: <Presentation Title> 40 of 70 bit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 sp1G1 sp1G2 sp1G3 sp1G4 sp2G1 sp2G2 sp2G3 sp2G4 sp3G1 sp3G2 sp3G3 sp3G4 sp4G1 sp4G2 sp4G3 sp4G4
41. Sensor Architecture PLL Registers SPI-2 SPI-1 Shutter controller (SC) Charge modulator drivers Macropixel array Column amplifiers and A/D converters Readout circuit Capture start/end and shutter pattern readout controller Shutter pattern memory (8b×32×16ch) Shift register array (8b×16ch) Drain controller CLK FETCH ADR[4:0] 16b 8b×16ch GSP1[4:1], GSP2[4:1], GSP3[4:1], GSP4[4:1] 16b TRG RDY SYNC SHLN[5:0] ENDRN SHLN[5:0] ENDRN Vertical scanner Horizontal scanner Digital image output CLK CLK High-speed part
42. Timing Chart for Single-Shot CLK TRG RDY SYNC ADR[4:0] FETCH GSP1[1] GSP2[2] End of capturing Cycle Start End 0 1 0 0 1st byte 2nd byte Standby Image capturing Standby
43. Chip Microphotograph Macropixel array Shutter controller Charge modulator drivers SPI-2 & registers Vertical scanner PLL Column amplifiers, A/D converters, readout circuit, and horizontal scanner
44. Specifications Comparison This work ISSCC’15 MDPI Sensors MDPI Sensors ISSCC’12 Technology 0.11μm CMOS FSI 0.11μm CMOS FSI 0.18μm CMOS FSI 0.13μm CMOS/CCD BSI 0.18μm CMOS FSI Chip size 7.0mmH× 9.3mmV 7.0mmH× 9.3mmV 4.8mmH× 4.8mmV - 15mmH× 24mmV (Macro)pixel size 22.4μmH× 22.4μmV 11.2μmH× 5.6μmV 70μmH×35μmV 12.73μmH×12.7 3μmV 32μmH× 32μmV Effective (sub)pixel count 212H×188V 320H×324V 50H×108V 576H×512V 400H×250V (Sub)pixel count per macro pixel/aperture 2H×2V 5H×3V - - - Tap count per (sub)pixel/aperture 4 1 + drain - - - Maximum shutter length 256b 128b - - - Number of frames in burst operation 12@1x, 32@2x 15@1x, 30@2x 368 (in-pixel), 184 (off-pixel) 10 248 Maximum burst frame rate 303Mfps 200Mfps 100Mfps (in-pixel), 125Mfps(off-pixel) 100Mfps 20Mfps Image readout frame rate 21fps 22fps N/A - 15kfps Power consumption 2.8W 1.62W N/A - 24W Multiple exposure Yes Yes No No No Compatibility with normal optics Yes No Yes Yes Yes
45. Experimental Setup for Single-Shot Filming Microscope (5x) Microscope (5x) RGB camera Developed CMOS image sensor PD for stop trigger 10x objective lens Pulse laser (λ=1064nm) Specimen (SPCC) Plasma image by the RGB camera Motorized stage
46. Emission Intensity Measured by PD 100 mV 20 ns 120 ns
47. Shutter for Non-Compression bit 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 sp1G1 sp1G2 sp1G3 sp1G4 sp2G1 sp2G2 sp2G3 sp2G4 sp3G1 sp3G2 sp3G3 sp3G4 sp4G1 sp4G2 sp4G3 sp4G4 Drain 3.29 ns
48. Shutter for 2x Compression 3.29 ns bit 1 2 3 4 5 6 7 8 9 101112131415161718192021 2 2 23242526272829303132 sp1G1 sp1G2 sp1G3 sp1G4 sp2G1 sp2G2 sp2G3 sp2G4 sp3G1 sp3G2 sp3G3 sp3G4 sp4G1 sp4G2 sp4G3 sp4G4
49. Measured Impulse Response: Non-Compression • PLL frequency: 303.75 MHz • Wavelength: 445nm • Pulse width: <80ps
50. Tap SP 3 4 1 2 3 4 1 2 3 4 Measured Impulse Response: 2x Compression Tap SP 1 2 1 2 3 4 1 2 3 4
51. Experimental Results of Single-Shot Filming 4 captured mosaic images
52. Experimental Results of Single-Shot Filming <<2x compressed>> Reproduced 32 images Magnified images 3.3ns Time <<Non-compressed>>
53. Experimental Setup for Light Reflection Filming Pulsed laser (λ=660nm, pulse width of 7ns) Mirror Developed CMOS image sensor Aperture Light source and camera Captured scene Panel (13.0m) Doll (8.0m) Pylon (6.0m) Pillar (4.0m) Lens Diffuser
54. Captured Compressed Images 4 captured mosaic images
55. Reproduced Movie of Light Reflection Panel (13.0m) Doll (8.0m) Pylon (6.0m) Pillar (4.0m) Reproduced movie
56. Decomposing Dual Path in TOF Pulsed laser (660 nm) Developed camera Mirror Letters “SU” Weak diffuser 3.30 m Diffuser Letters
57. Experimental result Captured 16 images Reproduced movie Developed camera Letters “SU” Weak diffuser 3.30 m
58. One-Pass Image Reproduction with DNN Courtesy of Prof. Nagahara, Osaka Univ. Compressed image Reproduced images by DNN
59. Motion Recognition from a Compressed Image 1/2 Courtesy of Prof. Nagahara, Osaka Univ. Scene Reconstructed video Decoder Reconstruction Action label Hand Waving CNN Action recognition Coded exposure image Capture
60. Motion Recognition from a Compressed Image 2/2 Video Coded Covering something with something Moving something down Real Input Model Top 1 Top 3 Top 5 Video (upper bound) C3D 71.0 88.0 88.0 Single image Coded exposure (Proposed) SVC2D 72.0 84.0 88.0 Long exposure C2D 20.0 40.0 52.0 Short exposure C2D 21.0 47.0 60.0 Courtesy of Prof. Nagahara, Osaka Univ. Video Compressed Covering something with something Moving something down
61.  Inner-product (or correlation) operation as low-level charge- domain signal processing is introduced to compress temporal optical signals.  Temporally compressive ultra-high-speed CMOS image sensor based on the inner-product operation has been fabricated.  Single-shot filming and multi-exposure-based transient imaging have been demonstrated at a burst frame rate of 303Mfps.  Extended depth range and separation of dual-path components have been demonstrated.  Applications of DNN such as one-path image reproduction and motion recognition from a compressed image were introduced. Summary
62. References 1/2  (CS) E. J. Cades and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Processing Magazine, Vol. 25, pp. 21-30 (2008).  (compressive video1) Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” ICCV (2011).  (compressive video2) T. Sonoda, H. Nagahara, K. Endo, Y. Sugiyama, and R. Taniguchi, “High-speed imaging using CMOS image sensor with quasi pixel-wise exposure,” ICCP (2016).  (compressive video3) Y. Luo, J. Jiang, M. Cai, and S. Mirabbasi, “CMOS computational camera with a two-tap coded exposure image sensor for single-shot spatial-temporal compressive sensing,” Optics Express, Vol. 27, pp. 31475-31489 (2019).  (spatial CS) Y. Oike and A. E. Gamal, “A 256x256 CMOS image sensors with ΔΣ-based single-shot compressed sensing,” IEEE ISSCC Dig. Tech. Papers, pp. 386-387(2012).  (UHS1) T. Arai, J. Yonai, T. Hayashida, H. Ohtake, H. van Kujik, and T. G. Etoh, “A 252-V/lux・s, 16.7-million- frames-per-second 312-kpixel back-side-illuminated ultrahigh-speed charge-coupled device,” IEEE Electron Devices, Vol. 60, pp. 3450-3458(2013).  (UHS2) Y. Tochigi, et al., “A global-shutter CMOS image sensor with readout speed of 1Tpixel/s burst and 780Mpixel/s continuous,” ISSCC Dig. Tech. Papers, pp. 382-383 (2012).  (UHS3) M. Suzuki, et al., “Over 100 Million Frames per Second 368 Frames Global Shutter Burst CMOS Image Sensor with In-Pixel Trench Capacitor Memory Array,” Proc. IISW, pp. 266-269 (2019).  (UHS4) T. Etoh, T. Okinaka, Y. Takano, K. Takehara, H. Nakano, K. Shimonomura, T. Ando, N. Ngo, Y. Kamakura, V. Dao, A. Nguyen, E. Charbon, C. Zhang, P. Moor, P. Goeschalckx, and L. Haspeslagh, “Light-in- flight imaging by a silicon image sensor: toward the theoretical highest frame rate,” MDPI Sensors, Vol. 19, Article 2247 (2019).
63. References 2/2  (LEFM1) S. Kawahito, G. Baek, Z. Li, S. Han, M. Seo, K. Yasutomi, and K. Kagawa, “CMOS lock-in pixel image sensors with lateral electric field control for time-resolved imaging,” Int'l Image Sensor Workshop, pp. 361-364 (2013).  (LEFM2) M. Seo, K. Kagawa, K. Yasutomi, T. Takasawa, Y. Kawata, N. Teranishi, Z. Li, I.A. Halin, and S. Kawahito, “10.8ps-time-resolution 256x512 image sensor with 2-tap true-CDS lock-in pixels for fluorescence lifetime imaging,” IEEE ISSCC Dig. Tech. Papers, pp. 198-199 (2015).  (LEFM3) M. Seo, Y. Shirakawa, Y. Masuda, Y. Kawata, K. Kagawa, K. Yasutomi, S. Kawahito, “A programmable sub-nanosecond time-gated 4-tap lock-in pixel CMOS image sensor for real-time fluorescence lifetime imaging microscopy,” ISSCC Dig. Tech. Papers, pp. 70-71 (2017).  (LEFM4) Y. Shirakawa, K. Yasutomi, K. Kagawa, S. Aoyama, and S. Kawahito, “An 8-tap CMOS lock-in pixel image sensor for short-pulse time-of-flight measurements,” MDPI Sensors, Vol. 20, Article 1040 (2020).  (Our work1) F. Mochizuki, K. Kagawa, S. Okihara, M. Seo, Z. Bo, T. Takasawa, K. Yasutomi, and S. Kawahito, “Single-shot 200Mfps 5x3-aperture compressive CMOS imager,” IEEE ISSCC Dig. Tech. Papers, pp. 116- 117(2015).  (Our work2) K. Kagawa, T. Kokado, Y. Sato, F. Mochizuki, H. Nagahara, T. Takasawa, K. Yasutomi, and S. Kawahito, “Multi-tap Macro-Pixel Based Compressive Ultra-High-Speed CMOS Image Sensor,” Proc. IISW, pp. 270-273 (2019).  (CS action recognition) T. Okawara, M. Yoshida, H. Nagahara, and Y. Yagi, “Action recognition from a single coded image,” ICCP (2020).  (DNN compressive video) M. Yoshida, A. Torii, M. Okutomi, K. Endo, Y. Sugiyama, R. Taniguchi, and H. Nagahara, “Joint optimization for compressive video sensing and reconstruction under hardware constraints,” ECCV (2018).