Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization Kenta Yamamoto1 /
Yuki Koyama2 / Yoichi Ochiai1 1 University of Tsukuba, Digital Nature Group 2 National Institute of Advanced Industrial Science and Technology (AIST)

2 Overview of Our Proposal 5SBEJUJPOBMEFTJHONFUIPE 1SPQPTFEGSBNFXPSL w /PWFMEFTJHOGSBNFXPSLXJUIPVUNBOVBMBEKVTUNFOUT w
*OUVJUJWFEFTJHOPQFSBUJPOJODMVEJOHQBJOUJOHPOTIPUT w .BOVBMBEKVTUNFOUTBSFSFBMMZIFBWZQSPDFTT w 5SJBMBOEFSSPSQSPDFTTUBLFTNVDIUJNF

3 Related Work Schoeneman, Chris, et al. "Painting with light."
Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 1993. Murmann, Lukas, et al. "Computational bounce flash for indoor portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9. Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial robots." Proceedings of the Workshop on Computational Aesthetics. 2014. Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light Probe." UIST. 2019. Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted image compositing for photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1. JO$(XPSME "VUPNBUJD'MBTI 1PTUQSPDFTTJOH BU$BQUVSFUJNFTVQQPSU %SPOFMJHIUJOH

4 Lighting Design Interface in CG World Schoeneman, Chris, et
al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 1993. Pellacini, Fabio, Parag Tole, and Donald P. Greenberg. "A user interface for interactive cinematic shadow design." ACM Transactions on Graphics (TOG) 21.3 (2002): 563-566. 1FMMBDJOJFUBM<5P(> 4DIPFOFNBOFUBM<> 8FUBSHFUSFBMXPSMEMJHIUJOHGPSUIFQIPUPHSBQIZXJUIWBSJPVTMJHIUJOHEFWJDFT

5 Lighting Design in Real World (automatic lighting) Murmann, Lukas,
et al. "Computational bounce flash for indoor portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9. Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial robots." Proceedings of the Workshop on Computational Aesthetics. 2014. .VSNBOOFUBM<5P(> 4SJLBOUIFUBM<> 0VSHPBMJTBOJOUFSBDUJWFGSBNFXPSLUIBUSF fl FDUTUIFQIPUPHSBQIFSTQSFGFSFODFT

6 Lighting Design in Real World (at capture-time) -&FUBM<6*45> Jane,
L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light Probe." UIST. 2019. 8FNBOJQVMBUFWBSJPVTMJHIUJOHEFWJDFTBOEUBLFBIVNBOJOUIFMPPQBQQSPBDIUPPQUJNJ[FGPSUIFQSFGFSFODF

7 Lighting Design in Real World (post-processing; re-lighting) Boyadzhiev, Ivaylo,
Sylvain Paris, and Kavita Bala. "User-assisted image compositing for photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1. #PZBE[IJFWFUBM<5P(> Sun, Tiancheng, et al. "Single image portrait relighting." ACM Trans. Graph. 38.4 (2019): 79-1. 4VOFUBM< 5P(> 8FUBSHFUlDBQUVSFUJNFz XIJDIBMMPXTBSUJGBDUGSFFIJHIFTURVBMJUZSFTVMUT

Our Proposed Framework

9 Overview of our framework 1. Selection 2. Painting 3.
Search for Next Options Photographer-in-the-Loop Photographer Part System Part w 1IPUPHSBQIFS1BSU4FMFDUJPOBOE1BJOUJOH w 4ZTUFN1BSU4FBSDI/FYU0QUJPOT 8FEF fi OFUIJTMPPQQSPDFTTBT QIPUPHSBQIFSJOUIFMPPQQSPDFTT

10 Human-in-the-loop Design Optimization (Human Evaluator) 1. Selection 2. Painting
3. Search for Next Options Photographer-in-the-Loop Photographer Part System Part 5IJTSFTFBSDIJTPOFPGUIF)VNBOJOUIFMPPQ )*5- %FTJHO0QUJNJ[BUJPO *O)*5-EFTJHOPQUJNJ[BUJPO IVNBOTBSFFWBMVBUPSTGPSTPMWJOHEFTJHOPQUJNJ[BUJPOQSPCMFNT   XJUIQFSDFQUVBMPCKFDUJWFGVODUJPO &WBMVBUPSXJUIQFSDFQUVBM PCKFDUJWFGVODUJPO )VNBO

%FTJHOPQUJNJ[BUJPODBOCFEF fi OFEBTUIFQBSBNFUFS PQUJNJ[BUJPO 'PSFYBNQMF UIFSFJTUIFBSSBOHFNFOUUBTLPGB QIPUPHSBQIFOIBODFNFOU 5IFOVNCFSPGTMJEFSTJTUIFEJNFOTJPOPGUIFJOQVU WBSJBCMFT BOEUIFVTFSTFBSDIFTUIFPQUJNBMQBSBNFUFS
TFUPGUIFN *EPOUMJLFUIJTDPMPS $IBOHFXIJUFCBMBODFUPCMVF 11 Human-in-the-loop Design Optimization (Parameter Optimization) FYQIPUPHSBQIFOIBODFNFOU )VNBO

8FEF fi OFUIFVTFSQSFGFSFODFBTUIFPCKFDUJWFGVODUJPO   IFSF XFDBMM(PPEOFTT'VODUJPOH Y 8FXBOUUPTPMWFUIFPQUJNJ[BUJPOQSPCMFN 12
Problem Formulation (parameter optimization) -JHIUJOHEFTJHODBOBMTPCFEF fi OFEBTBQBSBNFUFSPQUJNJ[BUJPO QSPCMFN -JHIUJOHQBSBNFUFST Y DBOCFEF fi OFEBTGPMMPXT   BMMQBSBNFUFSTBSFOPSNBMJ[FEXJUIJOd <YJ YJ > <YK YK > <YL YL > H Y H Y JTBCMBDLCPYGVODUJPO H Y JTBQFSDFQUVBMPCKFDUJWFGVODUJPO 8FDBOUPCTFSWFJUTGVODUJPOWBMVFTEJSFDUMZ

13 Bayesian Optimization (BO) #BZFTJBOPQUJNJ[BUJPO #0 JTXFMMLOPXOBTUIFTBNQMFFG fi DJFOUQBSBNFUFSTFBSDINFUIPE #0DBOCFBQQMJFEUPUIFCMBDLCPYPCKFDUJWFGVODUJPOIPXFWFS
  JUBTTVNFTUIBUXFDBOPCTFSWFUIFBCTPMVUFWBMVFTPGUIFCMBDLCPYGVODUJPO Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239. ,PZBNBFUBM<> 4UBOEBSE#0DBOUCFBQQMJFEEJSFDUMZUPIVNBOJOUIFMPPQ EFTJHOPQUJNJ[BUJPO )VNBOTBSFOPUHPPEBUBOTXFSJOHBCTPMVUF   WBMVFTBTTPDJBUFEXJUIJOQVUQBSBNFUFST

14 Preferential Bayesian Optimization (PBO) / Preferential Data Model 'PSUIF)*5-EFTJHOPQUJNJ[BUJPO
JOTUFBEPGHFUUJOHBCTPMVUF WBMVFT SFMBUJWFDPNQBSJTPOEBUBBSFVTFE ˠ1SFGFSFOUJBM#BZFTJBO0QUJNJ[BUJPO 1#0 ,PZBNBFUBM<> Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239. 4BNQMFEBUBPGDPNQBSJTPOEJ 'PSUIFDPNQBSJTPOXJUINVMUJQMFPQUJPOT XFBEPQU#SBEMFZ 5FSSZ-VDF #5- NPEFMBTGPMMPXT 8FJOGFSUIFWBMVFTPGH Y CZ."1FTUJNBUJPOVTJOHUIJTNPEFM

15 Preferential Bayesian Optimization in Our Framework #FDBVTFH Y JTBCMBDLCPYGVODUJPO
XFDBOOPUEJSFDUMZTPMWF UIJTPQUJNJ[BUJPOQSPCMFN *O#BZFTJBOPQUJNJ[BUJPO BDRVJTJUJPOGVODUJPO B JTVTFEUP EFUFSNJOFUIFOFYUTBNQMFTJOFBDIJUFSBUJPO <YJ YJ > <YK YK > <YL YL > H Y

16 Preferential Bayesian Optimization in Our Framework <YJ YJ
> <YK YK > <YL YL > H Y 5IFBDRVJTJUJPOGVODUJPOCBMBODFTFYQMPSBUJPO GBWPSVOFYQMPSFE BSFBT BOEFYQMPJUBUJPO GBWPSIJHIFYQFDUBUJPOBSFBT 4FBSDIJOHVOFYQMPSFEBSFB 5IFDPMPSNBQSFQSFTFOUTBDRVJTJUJPOGVODUJPOWBMVFTUIFSFEQPJOU SFQSFTFOUTUIFOFYUTBNQMF

*GPOMZUIFVTFSTTFMFDUJPOJOGPSNBUJPOJTVTFE UIFTUBOEBSEBDRVJTJUJPOGVODUJPOJTFOPVHI 8FJOUSPEVDFBOBEWBODFEBDRVJTJUJPOGVODUJPOGPSVTJOHUIFVTFSTSPVHIQBJOUJOH *JHVJEF BTBOBEEJUJPOBMHVJEF 17 PBO with Paint-based Guidance
in Our Framework Selection Paint #Iteration 3 #Iteration 1 Selection Paint a i (x) G(I(x), I i guide) a i guide(x;I i guide) a i (x) G(I(x), I i guide) a i guide(x;I i guide) Selection Paint #Iteration 2 a i (x) G(I(x), I i guide) a i guide(x;I i guide)

18 5IJTJTUIFFYUFOEFEWFSTJPOPG<$IPOHFUBM> "OFXUFSNGPSUIFHVJEFQBJOUJOH ( JTBEEFE Chong, Toby, et al. "Interactive
Optimization of Generative Image Modelling using Sequential Subspace Search and Content-based Guidance." Computer Graphics Forum. Vol. 40. No. 1. 2021. PBO with Paint-based Guidance in Our Framework 5PPQUJNJ[FGPSUIFOFXBDRVJTJUJPOGVODUJPO XFOFFEUPFWBMVBUF* Y GPSBSCJUSBSZYIPXFWFS QIZTJDBMMZUBLJOHBQIPUPFBDIUJNF UBLFT UPPNVDIUJNF BOE JTOPUEJGGFSFOUJBCMF ˠ-JHIUJOH1SFEJDUJPO OFYUTMJEF TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU UIFHSBEJFOUPGUIFBDRVJTJUJPOGVODUJPO

TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU 19 8FBEPQU(BVTTJBOQSPDFTTSFHSFTTPS (13 POFPGTBNQMFCBTFE QSFEJDUJPONFUIPET BTBEJGGFSFOUJBCMFMJHIUJOHQSFEJDUJPONFUIPE 5PUSBJOUIJTSFHSFTTJPONPEFM
XFHBUIFSUSBJOJOHEBUBCZBVUPNBUJDBMMZ UBLJOHQIPUPTXJUISBOEPNWBSJBCMFTCFGPSFIBOE dNJO 8FDIPTF(13GPSJUTTJNQMJDJUZ Lighting Prediction in Our Framework Ground Truth Prediction

20 Overall Workflow $BQUVSJOHUSBJOJOHEBUBGPSMJHIUJOHQSFEJDUJPO BOE $POTUSVDUJOHSFHSFTTPSGPSMJHIUJOHQSFEJDUJPO

21 Overall Workflow $IPPTFJOJUJBMSBOEPNMJHIUJOHDPO fi HVSBUJPOTGPS fi STUJUFSBUJPO

22 Overall Workflow 6TFSTFMFDUJPOBOEQBJOUJOH

23 Overall Workflow 5IFTZTUFNTFBSDIFTOFYUPQUJPOTVTJOH#BZFTJBOPQUJNJ[BUJPO

Live Demonstration (simulated virtual environment; not physical environment)

Result & Evaluation

26 Result B 1PTJUJPO % C 3FqFDUPSBOE -JHIU1PTJUJPO % D
1PTJUJPOBOE *OUFOTJUZ % F 1IZTJDBM 3PCPU"SNT % *UFSBUJPO *UFSBUJPO *UFSBUJPO E -JHIUT1PTJUJPO BOE$PMPS % 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 8FIBWFEFTJHOTFTTJPOTSFTVMUTVOEFSWBSJPVT DPOEJUJPOT WBSJPVTJOQVUWBSJBCMFT 'PSFYBNQMF w UIFEJNFOTJPOPGUIFJOQVUWBSJBCMFT w WBSJPVTMJHIUJOHQBSBNFUFST w TJNVMBUJPOPSQIZTJDBMFOWJSPONFOU

27 Technical Evaluation Target Paint (a) 3D (Light Position) (b)
5D (Light and Reflector Position) Target Paint (c) 8D (2 Lights Position and Power) Target Paint (d) 12D (2 Lights Position and Color) Target Paint

28 User Study (a) Proposed Framework Paint Interface White Light
with Robot Arm Orange Light with Robot Arm Reflector with Robot Arm White Light with Tripod Orange Light with Tripod Reflector with Tripod (b) Traditional Framework Captured image on camera is displayed Shooting Objects Shooting Objects 8FDPOEVDUFEUIFVTFSTUVEZXJUIQBSUJDJQBOUT 8FQSFQBSFEUZQFTPGUIFGSBNFXPSLGPSUIFQIPUPHSBQIJDMJHIUJOHEFTJHO USBEJUJPOBM NBOVBMBEKVTUNFOUBOEPVSGSBNFXPSL

29 User Study 0VST .BOVBM (a) Questionnaires (b) Satisfaction Q
Q *OBMMRVFTUJPOT PVSQSPQPTFEGSBNFXPSLXBTFWBMVBUFECFUUFSUIBOUIF USBEJUJPOBMGSBNFXPSL &TQFDJBMMZ 2JTUIFNPTUJNQPSUBOURVFTUJPOGPSUIFWBMJEBUJPOPGPVS DPOUSJCVUJPOT *UJTJOEJDBUFEUIBUQFPQMFDBOCFDPNFGPDVTJOHPOUIFEFTJHOJOHUBTL

30 Discussion -JNJUBUJPOTPGPVSVTFSTUVEZ )JHIFSEJNFOTJPOBMJUZ #0JTOPUHPPEBUIJHIEJNFOTJPOBMQSPCMFNT "DDVSBDZPG-JHIUJOH1SFEJDUJPO *UTFOPVHIGPSSPVHIHVJEJOH
CVUOPUWFSZBDDVSBUF *OUFHSBUJPOXJUIFYQFSULOPXMFEHFBOEQBUUFSOT *ODPSQPSBUJOHNPSFQBSBNFUFSTGPSQIPUPHSBQIZ &YUFOEJOHGFBTJCMFMJHIUJOHEFTJHOT &TQFDJBMMZ QIZTJDBMTFUVQJTIFBWJMZSFTUSJDUFECZIBSEXBSFTFUVQ 5PXBSETWJSUVBMQSPEVDUJPOBOEDJOFNBUPHSBQIZ

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization Kenta Yamamoto1 /
Yuki Koyama2 / Yoichi Ochiai1 1 University of Tsukuba, Digital Nature Group 2 National Institute of Advanced Industrial Science and Technology (AIST)

Photographic Lighting Design with Photographer-...

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Digital Nature Group

More Decks by Digital Nature Group

Other Decks in Research

Featured

Transcript