Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Slide 1

Slide 1 text

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization Kenta Yamamoto1 / Yuki Koyama2 / Yoichi Ochiai1 1 University of Tsukuba, Digital Nature Group 2 National Institute of Advanced Industrial Science and Technology (AIST)

Slide 2

Slide 2 text

2 Overview of Our Proposal 5SBEJUJPOBMEFTJHONFUIPE 1SPQPTFEGSBNFXPSL w /PWFMEFTJHOGSBNFXPSLXJUIPVUNBOVBMBEKVTUNFOUT w *OUVJUJWFEFTJHOPQFSBUJPOJODMVEJOHQBJOUJOHPOTIPUT w .BOVBMBEKVTUNFOUTBSFSFBMMZIFBWZQSPDFTT w 5SJBMBOEFSSPSQSPDFTTUBLFTNVDIUJNF

Slide 3

Slide 3 text

3 Related Work Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 1993. Murmann, Lukas, et al. "Computational bounce flash for indoor portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9. Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial robots." Proceedings of the Workshop on Computational Aesthetics. 2014. Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light Probe." UIST. 2019. Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted image compositing for photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1. JO$(XPSME "VUPNBUJD'MBTI 1PTUQSPDFTTJOH BU$BQUVSFUJNFTVQQPSU %SPOFMJHIUJOH

Slide 4

Slide 4 text

4 Lighting Design Interface in CG World Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 1993. Pellacini, Fabio, Parag Tole, and Donald P. Greenberg. "A user interface for interactive cinematic shadow design." ACM Transactions on Graphics (TOG) 21.3 (2002): 563-566. 1FMMBDJOJFUBM<5P(> 4DIPFOFNBOFUBM<> 8FUBSHFUSFBMXPSMEMJHIUJOHGPSUIFQIPUPHSBQIZXJUIWBSJPVTMJHIUJOHEFWJDFT

Slide 5

Slide 5 text

5 Lighting Design in Real World (automatic lighting) Murmann, Lukas, et al. "Computational bounce flash for indoor portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9. Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial robots." Proceedings of the Workshop on Computational Aesthetics. 2014. .VSNBOOFUBM<5P(> 4SJLBOUIFUBM<> 0VSHPBMJTBOJOUFSBDUJWFGSBNFXPSLUIBUSF fl FDUTUIFQIPUPHSBQIFSTQSFGFSFODFT

Slide 6

Slide 6 text

6 Lighting Design in Real World (at capture-time) -&FUBM<6*45> Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light Probe." UIST. 2019. 8FNBOJQVMBUFWBSJPVTMJHIUJOHEFWJDFTBOEUBLFBIVNBOJOUIFMPPQBQQSPBDIUPPQUJNJ[FGPSUIFQSFGFSFODF

Slide 7

Slide 7 text

7 Lighting Design in Real World (post-processing; re-lighting) Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted image compositing for photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1. #PZBE[IJFWFUBM<5P(> Sun, Tiancheng, et al. "Single image portrait relighting." ACM Trans. Graph. 38.4 (2019): 79-1. 4VOFUBM< 5P(> 8FUBSHFUlDBQUVSFUJNFz XIJDIBMMPXTBSUJGBDUGSFFIJHIFTURVBMJUZSFTVMUT

Slide 8

Slide 8 text

Our Proposed Framework

Slide 9

Slide 9 text

9 Overview of our framework 1. Selection 2. Painting 3. Search for Next Options Photographer-in-the-Loop Photographer Part System Part w 1IPUPHSBQIFS1BSU4FMFDUJPOBOE1BJOUJOH w 4ZTUFN1BSU4FBSDI/FYU0QUJPOT 8FEF fi OFUIJTMPPQQSPDFTTBT QIPUPHSBQIFSJOUIFMPPQQSPDFTT

Slide 10

Slide 10 text

10 Human-in-the-loop Design Optimization (Human Evaluator) 1. Selection 2. Painting 3. Search for Next Options Photographer-in-the-Loop Photographer Part System Part 5IJTSFTFBSDIJTPOFPGUIF)VNBOJOUIFMPPQ )*5- %FTJHO0QUJNJ[BUJPO *O)*5-EFTJHOPQUJNJ[BUJPO IVNBOTBSFFWBMVBUPSTGPSTPMWJOHEFTJHOPQUJNJ[BUJPOQSPCMFNT   XJUIQFSDFQUVBMPCKFDUJWFGVODUJPO &WBMVBUPSXJUIQFSDFQUVBM PCKFDUJWFGVODUJPO )VNBO

Slide 11

Slide 11 text

%FTJHOPQUJNJ[BUJPODBOCFEF fi OFEBTUIFQBSBNFUFS PQUJNJ[BUJPO 'PSFYBNQMF UIFSFJTUIFBSSBOHFNFOUUBTLPGB QIPUPHSBQIFOIBODFNFOU 5IFOVNCFSPGTMJEFSTJTUIFEJNFOTJPOPGUIFJOQVU WBSJBCMFT BOEUIFVTFSTFBSDIFTUIFPQUJNBMQBSBNFUFS TFUPGUIFN *EPOUMJLFUIJTDPMPS $IBOHFXIJUFCBMBODFUPCMVF 11 Human-in-the-loop Design Optimization (Parameter Optimization) FYQIPUPHSBQIFOIBODFNFOU )VNBO

Slide 12

Slide 12 text

8FEF fi OFUIFVTFSQSFGFSFODFBTUIFPCKFDUJWFGVODUJPO   IFSF XFDBMM(PPEOFTT'VODUJPOH Y 8FXBOUUPTPMWFUIFPQUJNJ[BUJPOQSPCMFN 12 Problem Formulation (parameter optimization) -JHIUJOHEFTJHODBOBMTPCFEF fi OFEBTBQBSBNFUFSPQUJNJ[BUJPO QSPCMFN -JHIUJOHQBSBNFUFST Y DBOCFEF fi OFEBTGPMMPXT   BMMQBSBNFUFSTBSFOPSNBMJ[FEXJUIJOd H Y H Y JTBCMBDLCPYGVODUJPO H Y JTBQFSDFQUVBMPCKFDUJWFGVODUJPO 8FDBOUPCTFSWFJUTGVODUJPOWBMVFTEJSFDUMZ

Slide 13

Slide 13 text

13 Bayesian Optimization (BO) #BZFTJBOPQUJNJ[BUJPO #0 JTXFMMLOPXOBTUIFTBNQMFFG fi DJFOUQBSBNFUFSTFBSDINFUIPE #0DBOCFBQQMJFEUPUIFCMBDLCPYPCKFDUJWFGVODUJPOIPXFWFS   JUBTTVNFTUIBUXFDBOPCTFSWFUIFBCTPMVUFWBMVFTPGUIFCMBDLCPYGVODUJPO Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239. ,PZBNBFUBM<> 4UBOEBSE#0DBOUCFBQQMJFEEJSFDUMZUPIVNBOJOUIFMPPQ EFTJHOPQUJNJ[BUJPO )VNBOTBSFOPUHPPEBUBOTXFSJOHBCTPMVUF   WBMVFTBTTPDJBUFEXJUIJOQVUQBSBNFUFST

Slide 14

Slide 14 text

14 Preferential Bayesian Optimization (PBO) / Preferential Data Model 'PSUIF)*5-EFTJHOPQUJNJ[BUJPO JOTUFBEPGHFUUJOHBCTPMVUF WBMVFT SFMBUJWFDPNQBSJTPOEBUBBSFVTFE ˠ1SFGFSFOUJBM#BZFTJBO0QUJNJ[BUJPO 1#0 ,PZBNBFUBM<> Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239. 4BNQMFEBUBPGDPNQBSJTPOEJ 'PSUIFDPNQBSJTPOXJUINVMUJQMFPQUJPOT XFBEPQU#SBEMFZ 5FSSZ-VDF #5- NPEFMBTGPMMPXT 8FJOGFSUIFWBMVFTPGH Y CZ."1FTUJNBUJPOVTJOHUIJTNPEFM

Slide 15

Slide 15 text

15 Preferential Bayesian Optimization in Our Framework #FDBVTFH Y JTBCMBDLCPYGVODUJPO XFDBOOPUEJSFDUMZTPMWF UIJTPQUJNJ[BUJPOQSPCMFN *O#BZFTJBOPQUJNJ[BUJPO BDRVJTJUJPOGVODUJPO B JTVTFEUP EFUFSNJOFUIFOFYUTBNQMFTJOFBDIJUFSBUJPO H Y

Slide 16

Slide 16 text

16 Preferential Bayesian Optimization in Our Framework H Y 5IFBDRVJTJUJPOGVODUJPOCBMBODFTFYQMPSBUJPO GBWPSVOFYQMPSFE BSFBT BOEFYQMPJUBUJPO GBWPSIJHIFYQFDUBUJPOBSFBT 4FBSDIJOHVOFYQMPSFEBSFB 5IFDPMPSNBQSFQSFTFOUTBDRVJTJUJPOGVODUJPOWBMVFTUIFSFEQPJOU SFQSFTFOUTUIFOFYUTBNQMF

Slide 17

Slide 17 text

*GPOMZUIFVTFSTTFMFDUJPOJOGPSNBUJPOJTVTFE UIFTUBOEBSEBDRVJTJUJPOGVODUJPOJTFOPVHI 8FJOUSPEVDFBOBEWBODFEBDRVJTJUJPOGVODUJPOGPSVTJOHUIFVTFSTSPVHIQBJOUJOH *JHVJEF BTBOBEEJUJPOBMHVJEF 17 PBO with Paint-based Guidance in Our Framework Selection Paint #Iteration 3 #Iteration 1 Selection Paint a i (x) G(I(x), I i guide) a i guide(x;I i guide) a i (x) G(I(x), I i guide) a i guide(x;I i guide) Selection Paint #Iteration 2 a i (x) G(I(x), I i guide) a i guide(x;I i guide)

Slide 18

Slide 18 text

18 5IJTJTUIFFYUFOEFEWFSTJPOPG<$IPOHFUBM> "OFXUFSNGPSUIFHVJEFQBJOUJOH ( JTBEEFE Chong, Toby, et al. "Interactive Optimization of Generative Image Modelling using Sequential Subspace Search and Content-based Guidance." Computer Graphics Forum. Vol. 40. No. 1. 2021. PBO with Paint-based Guidance in Our Framework 5PPQUJNJ[FGPSUIFOFXBDRVJTJUJPOGVODUJPO XFOFFEUPFWBMVBUF* Y GPSBSCJUSBSZYIPXFWFS QIZTJDBMMZUBLJOHBQIPUPFBDIUJNF UBLFT UPPNVDIUJNF BOE JTOPUEJGGFSFOUJBCMF ˠ-JHIUJOH1SFEJDUJPO OFYUTMJEF TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU UIFHSBEJFOUPGUIFBDRVJTJUJPOGVODUJPO

Slide 19

Slide 19 text

TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU 19 8FBEPQU(BVTTJBOQSPDFTTSFHSFTTPS (13 POFPGTBNQMFCBTFE QSFEJDUJPONFUIPET BTBEJGGFSFOUJBCMFMJHIUJOHQSFEJDUJPONFUIPE 5PUSBJOUIJTSFHSFTTJPONPEFM XFHBUIFSUSBJOJOHEBUBCZBVUPNBUJDBMMZ UBLJOHQIPUPTXJUISBOEPNWBSJBCMFTCFGPSFIBOE dNJO 8FDIPTF(13GPSJUTTJNQMJDJUZ Lighting Prediction in Our Framework Ground Truth Prediction

Slide 20

Slide 20 text

20 Overall Workflow $BQUVSJOHUSBJOJOHEBUBGPSMJHIUJOHQSFEJDUJPO BOE $POTUSVDUJOHSFHSFTTPSGPSMJHIUJOHQSFEJDUJPO

Slide 21

Slide 21 text

21 Overall Workflow $IPPTFJOJUJBMSBOEPNMJHIUJOHDPO fi HVSBUJPOTGPS fi STUJUFSBUJPO

Slide 22

Slide 22 text

22 Overall Workflow 6TFSTFMFDUJPOBOEQBJOUJOH

Slide 23

Slide 23 text

23 Overall Workflow 5IFTZTUFNTFBSDIFTOFYUPQUJPOTVTJOH#BZFTJBOPQUJNJ[BUJPO

Slide 24

Slide 24 text

Live Demonstration (simulated virtual environment; not physical environment)

Slide 25

Slide 25 text

Result & Evaluation

Slide 26

Slide 26 text

26 Result B 1PTJUJPO % C 3FqFDUPSBOE -JHIU1PTJUJPO % D 1PTJUJPOBOE *OUFOTJUZ % F 1IZTJDBM 3PCPU"SNT % *UFSBUJPO *UFSBUJPO *UFSBUJPO E -JHIUT1PTJUJPO BOE$PMPS % 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 8FIBWFEFTJHOTFTTJPOTSFTVMUTVOEFSWBSJPVT DPOEJUJPOT WBSJPVTJOQVUWBSJBCMFT 'PSFYBNQMF w UIFEJNFOTJPOPGUIFJOQVUWBSJBCMFT w WBSJPVTMJHIUJOHQBSBNFUFST w TJNVMBUJPOPSQIZTJDBMFOWJSPONFOU

Slide 27

Slide 27 text

27 Technical Evaluation Target Paint (a) 3D (Light Position) (b) 5D (Light and Reflector Position) Target Paint (c) 8D (2 Lights Position and Power) Target Paint (d) 12D (2 Lights Position and Color) Target Paint

Slide 28

Slide 28 text

28 User Study (a) Proposed Framework Paint Interface White Light with Robot Arm Orange Light with Robot Arm Reflector with Robot Arm White Light with Tripod Orange Light with Tripod Reflector with Tripod (b) Traditional Framework Captured image on camera is displayed Shooting Objects Shooting Objects 8FDPOEVDUFEUIFVTFSTUVEZXJUIQBSUJDJQBOUT 8FQSFQBSFEUZQFTPGUIFGSBNFXPSLGPSUIFQIPUPHSBQIJDMJHIUJOHEFTJHO USBEJUJPOBM NBOVBMBEKVTUNFOUBOEPVSGSBNFXPSL

Slide 29

Slide 29 text

29 User Study 0VST .BOVBM (a) Questionnaires (b) Satisfaction Q Q *OBMMRVFTUJPOT PVSQSPQPTFEGSBNFXPSLXBTFWBMVBUFECFUUFSUIBOUIF USBEJUJPOBMGSBNFXPSL &TQFDJBMMZ 2JTUIFNPTUJNQPSUBOURVFTUJPOGPSUIFWBMJEBUJPOPGPVS DPOUSJCVUJPOT *UJTJOEJDBUFEUIBUQFPQMFDBOCFDPNFGPDVTJOHPOUIFEFTJHOJOHUBTL

Slide 30

Slide 30 text

30 Discussion -JNJUBUJPOTPGPVSVTFSTUVEZ )JHIFSEJNFOTJPOBMJUZ #0JTOPUHPPEBUIJHIEJNFOTJPOBMQSPCMFNT "DDVSBDZPG-JHIUJOH1SFEJDUJPO *UTFOPVHIGPSSPVHIHVJEJOH CVUOPUWFSZBDDVSBUF *OUFHSBUJPOXJUIFYQFSULOPXMFEHFBOEQBUUFSOT *ODPSQPSBUJOHNPSFQBSBNFUFSTGPSQIPUPHSBQIZ &YUFOEJOHGFBTJCMFMJHIUJOHEFTJHOT &TQFDJBMMZ QIZTJDBMTFUVQJTIFBWJMZSFTUSJDUFECZIBSEXBSFTFUVQ 5PXBSETWJSUVBMQSPEVDUJPOBOEDJOFNBUPHSBQIZ