Unphotogenic Light - 2018 SID Display Week Symposium (Oral presentation by Ippei Suzuki)

This slide was presented in Session 70: Projection: Screen Technology at 2018 SID Display Week Symposium.
http://www.displayweek.org

“Unphotogenic Light”: Evaluation and Detail of the High-Speed Projection Method to Prevent Secret Photography with Small Cameras (70-1)
Ippei Suzuki, Shinnosuke Ando, Yoichi Ochiai
Digital Nature Group, University of Tsukuba, Japan
Pixie Dust Technologies, Inc., Japan
https://onlinelibrary.wiley.com/loi/21680159

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2017/07/unphotogenic-light/

【Project movie】
https://www.youtube.com/watch?v=sCrAFTnqnkQ

【Presenter】
Ippei Suzuki (鈴木 一平)
University of Tsukuba, School of Informatics
College of Media Arts, Science and Technology
Digital Nature Group (Yoichi Ochiai)
http://1heisuzuki.tumblr.com

【Abstract】
Secret recordings using smartphones have become a serious problem for content providers. We present a new method to protect projected images and other information (i.e., “recordable content”) from secret photography by using high-speed projection. Our purpose in this study is to develop and implement a projection method that allows people to observe photographable objects, people, and events while preventing the same from being recorded by cameras and other recordable devices. To achieve this goal, we focus on the difference between human and camera vision systems. Unlike cameras, human beings cannot recognize the high-speed changes of light. We divide an image into two or more parts and project them in succession at a high frame rate such that a full image is made visible to human eyes. By contrast, cameras can only capture an incomplete frame.

1. Unphotogenic Light Evaluation and Detail of the High-Speed Projection Method to Prevent Secret Photography with Small Cameras Ippei Suzuki, Shinnosuke Ando, Yoichi Ochiai University of Tsukuba, Strategic Research Platform towards Digital Nature Powered by Pixie Dust Technologies, Inc. Session 70 - Projection: Screen Technology
2. 2
3. 3 Introduction Content Protection on the Screen Protecting images projected on a screen is a difficult problem
4. 4 Introduction Content Protection on the Screen Copy Contents is protected by data copy guard techniques People can take contents shown on the screenbut Protecting images projected on a screen is a difficult problem
5. Examples of the content captured by taking the screen 5 *This image is only for illustration purposes, NOT actual result from the video-sharing services.
6. Visible to Human Eyes Recordable by Cameras≒ 6
7. 7
8. 8 Introduction Practical Stories of the Proposed Method When you would like to protect this image from secret photography... Situation
9. 9 Introduction Practical Stories of the Proposed Method Solution A: Visible Watermarks / Protection Marks CONFIDENTIAL COPYRIGHT NO PHOTOS
10. CONFIDENTIAL COPYRIGHT NO PHOTOS 10 Introduction Practical Stories of the Proposed Method Bad Experience for Visitors… 🤔 Solution A: Visible Watermarks / Protection Marks
11. Security Check Area 11 Introduction Practical Stories of the Proposed Method Security Visitor Visitor’s Bag Solution B: Confiscate Camera Devices Confiscate Camera Devices
12. 12 Introduction Practical Stories of the Proposed Method Visitor Security Don’t use cameras!!! Solution B: Confiscate Camera Devices Security can orally warn overlooked cameras
13. 13 Introduction Practical Stories of the Proposed Method Visitor’s Pocket Visitor Secret Camera e.g., Pen-Shaped Small Camera They can take photos 😭 Solution B: Confiscate Camera Devices
14. 14 Introduction Target in This Study Our Target Kinds of the situation protecting the content Confiscate Camera Devices but contents are secretly captured CONFIDENTIAL COPYRIGHT NO PHOTOS Visible Watermarks not only camera, but also people
15. 15 Introduction Target in This Study Kinds of the camera which criminals may use Our Target “Spy Car Remote Control Camera” by DigitPedia Website (CC BY 2.0, Color Modified) “Sony Xperia Z3 smartphone” by John Jones (CC BY 2.0, Color Modified)“Google Glass” by Kārlis Dambrāns (CC BY 2.0, Background deleted) “Sony Alpha A77” by SkywalkerPL (CC BY 3.0) Not considered in ours Camera Camera Spy Camera Smart Glass Smartphone DSLR
16. 16 Introduction Goal Human eyes can see Cameras can NOT see = Unphotogenic
17. “Unphotogenic” — Looking non-attractive in photographs or on film — Not appear in photos as they really are in real life 17
18. 18
19. < Temporal Resolution 19
20. 20 Related Work VRCodes [2]Detection of Imperceptible On-Screen Markers with Unsynchronized Cameras [1] [1]
 [2] Invisible Markers on Screen Sampaio Luiz, Yamada Yoshio, Yamamoto Goshiro, Taketomi Takafumi, Sandor Christian, and Kato Hirokazu. 
 Detection of imperceptible on-screen markers with unsynchronized cameras. IPSJ SIG Notes. CVIM, 2015(64):1–4, jan 2015. Grace Woo, Andy Lippman, and Ramesh Raskar. Vrcodes: Unobtrusive and active visual codes for interaction by exploiting rolling shutter. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), ISMAR ’12, pages 59–64, Washington, DC, USA, 2012. IEEE Computer Society.
21. 21 Afterimage An impression of a vivid sensation retained after the stimulus has ceased
22. 22 Method Approach
23. 23 Method Approach Divide Image Switch in High Refresh Rate
24. 24 Method Theory and Effect of the Proposed System
25. 25 Implementation Projector DLP® LightCrafter™ 4500 EVM Texas Instruments DMD 912 × 1140 diamond pixel 0.45-inch WXGA DMD Resolution Up to 1280 × 800 Light source RGB LED light engine
 with 150 lumen light output Max video speed Up to 120 Hz Max pattern speed Up to 4225 Hz binary pattern
 Up to 120 Hz 8-bit grayscale pattern
26. 26 210mm 297 mm Projector Camera Screen
27. 27 Evaluation Patterns of division Plasma effect Patterns of division in evaluation Reference: Lode's Computer Graphics Tutorial
 http://lodev.org/cgtutor/plasma.html
28. 28 Result Evaluation from Camera Vision DSLR Smartphone Variable Aperture Fixed Aperture SS 1/15, F1.8 Canon EOS Kiss X4 w/ SIGMA 17-50mm F2.8 EX DC OS HSM All photos were taken by ISO 100, 50mm. Apple iPhone 7 w/ App “ProShot” by Rise Up Games LLC.) All photos were taken by F/1.8, ISO 20, 28mm. SS 1/30, F1.8 SS 1/120, F1.8 SS 1/160, F1.8 **Focal lengths are the corresponding value for a 35-mm Film. SS 1/15, F18 SS 1/30, F13 SS 1/125, F5.6 SS 1/160, F5.6 *Pictographs of shutter speeds and apertures indicate the rough value. Smartphones cannot change the aperture Aperture (F-stop)Shutter speed
29. 29 Method Theory and Effect of the Proposed System Don’t use cameras!!! Combination of the human power and our system Picture which our system applied Visitors cannot take photos of the picture!
30. Global ShutterRolling Shutter (popular) 30 Result Evaluation from Camera Vision (other smartphones & cameras) Apple Inc. iPhone 7 G8142; Sony Mobile Communications Inc. Xperia XZ Premium 5040o; TCL Communication Ltd. Alcatel OneTouch Flint PRA-LX2; HUAWEI Technologies Co. HUAWEY nova lite DSC-W110, Sony Corp. Cyber-shot Not affected by the type of the image sensor *Photos are taken with the default application, automatic mode.
31. 31 Application Other Examples than Displaying Divided Images EXIT $10 Theme Park Photo by Ted Murphy (CC BY 2.0) Photo by Ted Murphy (CC BY 2.0) Result Overlay onto the photos sold near the exit of the attraction
32. 32 Application Other Examples than Displaying Divided Images Museum Result Overlay onto the artworks Projector Artwork
33. 33 Discussion & Future Work Robustness Reconstructable from multiple photographs? All parts of the original image are needed
34. 34 Discussion & Future Work Flicker *This video is for illustration purposes. When a saccade occurs on human's eyes or when human blink, people sometimes perceive flicker. Tradeoffs between the strength of the system’s resistance to flicker and an acceptable shutter speed (slowness). Not suitable for extended viewing of media? Optimization of patterns of division?
35. 35 Discussion & Future Work Quality of the Display The brightness of each pixel becomes 1/n of the normal projection. But we need enough brightness of the screen to affect our system into photograph. Normal Ours Time Brighter projectors? N times division 1/N brightness Observed
36. 36
37. 37 Contact: 1heisuzuki@digitalnature.slis.tsukuba.ac.jp Unphotogenic Light Evaluation and Detail of the High-Speed Projection Method to Prevent Secret Photography with Small Cameras Ippei Suzuki, Shinnosuke Ando, Yoichi Ochiai University of Tsukuba, Strategic Research Platform towards Digital Nature Powered by Pixie Dust Technologies, Inc. Questions? Ippei Suzuki Shinnosuke Ando Yoichi Ochiai

Transcript

1. Unphotogenic Light Evaluation and Detail of the High-Speed Projection Method to Prevent Secret Photography with Small Cameras Ippei Suzuki, Shinnosuke Ando, Yoichi Ochiai University of Tsukuba, Strategic Research Platform towards Digital Nature Powered by Pixie Dust Technologies, Inc. Session 70 - Projection: Screen Technology
3. 3 Introduction Content Protection on the Screen Protecting images projected on a screen is a difficult problem
4. 4 Introduction Content Protection on the Screen Copy Contents is protected by data copy guard techniques People can take contents shown on the screen but Protecting images projected on a screen is a difficult problem
5. Examples of the content captured by taking the screen 5 *This image is only for illustration purposes, NOT actual result from the video-sharing services.
6. Visible to Human Eyes Recordable by Cameras ≒ 6
8. 8 Introduction Practical Stories of the Proposed Method When you would like to protect this image from secret photography... Situation
9. 9 Introduction Practical Stories of the Proposed Method Solution A: Visible Watermarks / Protection Marks CONFIDENTIAL COPYRIGHT NO PHOTOS
10. CONFIDENTIAL COPYRIGHT NO PHOTOS 10 Introduction Practical Stories of the Proposed Method Bad Experience for Visitors… Solution A: Visible Watermarks / Protection Marks
11. Security Check Area 11 Introduction Practical Stories of the Proposed Method Security Visitor Visitor’s Bag Solution B: Confiscate Camera Devices Confiscate Camera Devices
12. 12 Introduction Practical Stories of the Proposed Method Visitor Security Don’t use cameras!!! Solution B: Confiscate Camera Devices Security can orally warn overlooked cameras
13. 13 Introduction Practical Stories of the Proposed Method Visitor’s Pocket Visitor Secret Camera e.g., Pen-Shaped Small Camera They can take photos Solution B: Confiscate Camera Devices
14. 14 Introduction Target in This Study Our Target Kinds of the situation protecting the content Confiscate Camera Devices but contents are secretly captured CONFIDENTIAL COPYRIGHT NO PHOTOS Visible Watermarks not only camera, but also people
15. 15 Introduction Target in This Study Kinds of the camera which criminals may use Our Target “Spy Car Remote Control Camera” by DigitPedia Website (CC BY 2.0, Color Modified) “Sony Xperia Z3 smartphone” by John Jones (CC BY 2.0, Color Modified) “Google Glass” by Kārlis Dambrāns (CC BY 2.0, Background deleted) “Sony Alpha A77” by SkywalkerPL (CC BY 3.0) Not considered in ours Camera Camera Spy Camera Smart Glass Smartphone DSLR
16. 16 Introduction Goal Human eyes can see Cameras can NOT see = Unphotogenic
17. “Unphotogenic” — Looking non-attractive in photographs or on film — Not appear in photos as they really are in real life 17
19. < Temporal Resolution 19
20. 20 Related Work VRCodes [2] Detection of Imperceptible On-Screen Markers with Unsynchronized Cameras [1] [1]
 [2] Invisible Markers on Screen Sampaio Luiz, Yamada Yoshio, Yamamoto Goshiro, Taketomi Takafumi, Sandor Christian, and Kato Hirokazu. 
 Detection of imperceptible on-screen markers with unsynchronized cameras. IPSJ SIG Notes. CVIM, 2015(64):1–4, jan 2015. Grace Woo, Andy Lippman, and Ramesh Raskar. Vrcodes: Unobtrusive and active visual codes for interaction by exploiting rolling shutter. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), ISMAR ’12, pages 59–64, Washington, DC, USA, 2012. IEEE Computer Society.
21. 21 Afterimage An impression of a vivid sensation retained after the stimulus has ceased
22. 22 Method Approach
23. 23 Method Approach Divide Image Switch in High Refresh Rate
24. 24 Method Theory and Effect of the Proposed System
25. 25 Implementation Projector DLP® LightCrafter™ 4500 EVM Texas Instruments DMD 912 × 1140 diamond pixel 0.45-inch WXGA DMD Resolution Up to 1280 × 800 Light source RGB LED light engine
 with 150 lumen light output Max video speed Up to 120 Hz Max pattern speed Up to 4225 Hz binary pattern
 Up to 120 Hz 8-bit grayscale pattern
26. 26 210 mm 297 mm Projector Camera Screen
27. 27 Evaluation Patterns of division Plasma effect Patterns of division in evaluation Reference: Lode's Computer Graphics Tutorial
 http://lodev.org/cgtutor/plasma.html
28. 28 Result Evaluation from Camera Vision DSLR Smartphone Variable Aperture Fixed Aperture SS 1/15, F1.8 Canon EOS Kiss X4 w/ SIGMA 17-50mm F2.8 EX DC OS HSM All photos were taken by ISO 100, 50mm. Apple iPhone 7 w/ App “ProShot” by Rise Up Games LLC.) All photos were taken by F/1.8, ISO 20, 28mm. SS 1/30, F1.8 SS 1/120, F1.8 SS 1/160, F1.8 **Focal lengths are the corresponding value for a 35-mm Film. SS 1/15, F18 SS 1/30, F13 SS 1/125, F5.6 SS 1/160, F5.6 *Pictographs of shutter speeds and apertures indicate the rough value. Smartphones cannot change the aperture Aperture (F-stop) Shutter speed
29. 29 Method Theory and Effect of the Proposed System Don’t use cameras!!! Combination of the human power and our system Picture which our system applied Visitors cannot take photos of the picture!
30. Global Shutter Rolling Shutter (popular) 30 Result Evaluation from Camera Vision (other smartphones & cameras) Apple Inc. iPhone 7 G8142; Sony Mobile Communications Inc. Xperia XZ Premium 5040o; TCL Communication Ltd. Alcatel OneTouch Flint PRA-LX2; HUAWEI Technologies Co. HUAWEY nova lite DSC-W110, Sony Corp. Cyber-shot Not affected by the type of the image sensor *Photos are taken with the default application, automatic mode.
31. 31 Application Other Examples than Displaying Divided Images EXIT $10 Theme Park Photo by Ted Murphy (CC BY 2.0) Photo by Ted Murphy (CC BY 2.0) Result Overlay onto the photos sold near the exit of the attraction
32. 32 Application Other Examples than Displaying Divided Images Museum Result Overlay onto the artworks Projector Artwork
33. 33 Discussion & Future Work Robustness Reconstructable from multiple photographs? All parts of the original image are needed
34. 34 Discussion & Future Work Flicker *This video is for illustration purposes. When a saccade occurs on human's eyes or when human blink, people sometimes perceive flicker. Tradeoffs between the strength of the system’s resistance to flicker and an acceptable shutter speed (slowness). Not suitable for extended viewing of media? Optimization of patterns of division?
35. 35 Discussion & Future Work Quality of the Display The brightness of each pixel becomes 1/n of the normal projection. But we need enough brightness of the screen to affect our system into photograph. Normal Ours Time Brighter projectors? N times division 1/N brightness Observed
37. 37 Contact: 1heisuzuki@digitalnature.slis.tsukuba.ac.jp Unphotogenic Light Evaluation and Detail of the High-Speed Projection Method to Prevent Secret Photography with Small Cameras Ippei Suzuki, Shinnosuke Ando, Yoichi Ochiai University of Tsukuba, Strategic Research Platform towards Digital Nature Powered by Pixie Dust Technologies, Inc. Questions? Ippei Suzuki Shinnosuke Ando Yoichi Ochiai