Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Transcript

1. Photographic Lighting Design with
   
   
   Photographer-in-the-Loop Bayesian Optimization
   Kenta Yamamoto1 / Yuki Koyama2 / Yoichi Ochiai1
   1 University of Tsukuba, Digital Nature Group
   
   
   2 National Institute of Advanced Industrial Science and Technology (AIST)
   

   View Slide

2. 2
   Overview of Our Proposal
   5SBEJUJPOBMEFTJHONFUIPE 1SPQPTFEGSBNFXPSL
   w /PWFMEFTJHOGSBNFXPSLXJUIPVUNBOVBMBEKVTUNFOUT
   w *OUVJUJWFEFTJHOPQFSBUJPOJODMVEJOHQBJOUJOHPOTIPUT
   w .BOVBMBEKVTUNFOUTBSFSFBMMZIFBWZQSPDFTT
   w 5SJBMBOEFSSPSQSPDFTTUBLFTNVDIUJNF
   

   View Slide

3. 3
   Related Work
   Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and
   interactive techniques. 1993.
   Murmann, Lukas, et al. "Computational bounce flash for indoor
   portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9.
   Srikanth, Manohar, Kavita Bala, and Frédo Durand.
   "Computational rim illumination with aerial robots." Proceedings
   of the Workshop on Computational Aesthetics. 2014.
   Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing
   Portrait Lighting at Capture-Time Using a 360 Camera as a Light
   Probe." UIST. 2019.
   Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted
   image compositing for photographic lighting." ACM Trans.
   Graph. 32.4 (2013): 36-1.
   JO$(XPSME "VUPNBUJD'MBTI
   1PTUQSPDFTTJOH
   BU$BQUVSFUJNFTVQQPSU
   %SPOFMJHIUJOH
   

   View Slide

4. 4
   Lighting Design Interface in CG World
   Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and interactive techniques.
   1993.
   Pellacini, Fabio, Parag Tole, and Donald P. Greenberg. "A user
   interface for interactive cinematic shadow design." ACM Transactions
   on Graphics (TOG) 21.3 (2002): 563-566.
   1FMMBDJOJFUBM<5P(>
   4DIPFOFNBOFUBM<>
   8FUBSHFUSFBMXPSMEMJHIUJOHGPSUIFQIPUPHSBQIZXJUIWBSJPVTMJHIUJOHEFWJDFT
   

   View Slide

5. 5
   Lighting Design in Real World (automatic lighting)
   Murmann, Lukas, et al. "Computational bounce flash for indoor portraits." ACM Transactions on
   Graphics (TOG) 35.6 (2016): 1-9.
   Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial
   robots." Proceedings of the Workshop on Computational Aesthetics. 2014.
   .VSNBOOFUBM<5P(>
   4SJLBOUIFUBM<>
   0VSHPBMJTBOJOUFSBDUJWFGSBNFXPSLUIBUSF
   fl
   FDUTUIFQIPUPHSBQIFSTQSFGFSFODFT
   

   View Slide

6. 6
   Lighting Design in Real World (at capture-time)
   -&FUBM<6*45>
   Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light
   Probe." UIST. 2019.
   8FNBOJQVMBUFWBSJPVTMJHIUJOHEFWJDFTBOEUBLFBIVNBOJOUIFMPPQBQQSPBDIUPPQUJNJ[FGPSUIFQSFGFSFODF
   

   View Slide

7. 7
   Lighting Design in Real World (post-processing; re-lighting)
   Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted image compositing for
   photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1.
   #PZBE[IJFWFUBM<5P(>
   Sun, Tiancheng, et al. "Single image portrait relighting." ACM Trans. Graph. 38.4 (2019): 79-1.
   4VOFUBM< 5P(>
   8FUBSHFUlDBQUVSFUJNFz XIJDIBMMPXTBSUJGBDUGSFFIJHIFTURVBMJUZSFTVMUT
   

   View Slide

8. Our Proposed Framework
   

   View Slide

9. 9
   Overview of our framework
   1. Selection 2. Painting 3. Search for Next Options
   Photographer-in-the-Loop
   Photographer Part System Part
   w 1IPUPHSBQIFS1BSU4FMFDUJPOBOE1BJOUJOH
   w 4ZTUFN1BSU4FBSDI/FYU0QUJPOT
   8FEF
   fi
   OFUIJTMPPQQSPDFTTBT
   QIPUPHSBQIFSJOUIFMPPQQSPDFTT
   

   View Slide

10. 10
    Human-in-the-loop Design Optimization (Human Evaluator)
    1. Selection 2. Painting 3. Search for Next Options
    Photographer-in-the-Loop
    Photographer Part System Part
    5IJTSFTFBSDIJTPOFPGUIF)VNBOJOUIFMPPQ )*5-
    %FTJHO0QUJNJ[BUJPO
    *O)*5-EFTJHOPQUJNJ[BUJPO IVNBOTBSFFWBMVBUPSTGPSTPMWJOHEFTJHOPQUJNJ[BUJPOQSPCMFNT
    

    XJUIQFSDFQUVBMPCKFDUJWFGVODUJPO
    &WBMVBUPSXJUIQFSDFQUVBM
    PCKFDUJWFGVODUJPO
    )VNBO
    

    View Slide

11. %FTJHOPQUJNJ[BUJPODBOCFEF
    fi
    OFEBTUIFQBSBNFUFS
    PQUJNJ[BUJPO
    'PSFYBNQMF UIFSFJTUIFBSSBOHFNFOUUBTLPGB
    QIPUPHSBQIFOIBODFNFOU
    5IFOVNCFSPGTMJEFSTJTUIFEJNFOTJPOPGUIFJOQVU
    WBSJBCMFT BOEUIFVTFSTFBSDIFTUIFPQUJNBMQBSBNFUFS
    TFUPGUIFN
    *EPOUMJLFUIJTDPMPS
    $IBOHFXIJUFCBMBODFUPCMVF
    11
    Human-in-the-loop Design Optimization (Parameter Optimization)
    FYQIPUPHSBQIFOIBODFNFOU
    )VNBO
    

    View Slide

12. 8FEF
    fi
    OFUIFVTFSQSFGFSFODFBTUIFPCKFDUJWFGVODUJPO
    

    IFSF XFDBMM(PPEOFTT'VODUJPOH Y
    
    
    8FXBOUUPTPMWFUIFPQUJNJ[BUJPOQSPCMFN
    12
    Problem Formulation (parameter optimization)
    -JHIUJOHEFTJHODBOBMTPCFEF
    fi
    OFEBTBQBSBNFUFSPQUJNJ[BUJPO
    QSPCMFN
    -JHIUJOHQBSBNFUFST Y DBOCFEF
    fi
    OFEBTGPMMPXT
    

    BMMQBSBNFUFSTBSFOPSNBMJ[FEXJUIJOd
    
    
    
    
    H Y
    
    H Y
    JTBCMBDLCPYGVODUJPO
    H Y
    JTBQFSDFQUVBMPCKFDUJWFGVODUJPO
    8FDBOUPCTFSWFJUTGVODUJPOWBMVFTEJSFDUMZ
    

    View Slide

13. 13
    Bayesian Optimization (BO)
    #BZFTJBOPQUJNJ[BUJPO #0
    JTXFMMLOPXOBTUIFTBNQMFFG
    fi
    DJFOUQBSBNFUFSTFBSDINFUIPE
    #0DBOCFBQQMJFEUPUIFCMBDLCPYPCKFDUJWFGVODUJPOIPXFWFS
    

    JUBTTVNFTUIBUXFDBOPCTFSWFUIFBCTPMVUFWBMVFTPGUIFCMBDLCPYGVODUJPO
    Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239.
    ,PZBNBFUBM<>
    4UBOEBSE#0DBOUCFBQQMJFEEJSFDUMZUPIVNBOJOUIFMPPQ
    EFTJHOPQUJNJ[BUJPO
    )VNBOTBSFOPUHPPEBUBOTXFSJOHBCTPMVUF
    

    WBMVFTBTTPDJBUFEXJUIJOQVUQBSBNFUFST
    

    View Slide

14. 14
    Preferential Bayesian Optimization (PBO) / Preferential Data Model
    'PSUIF)*5-EFTJHOPQUJNJ[BUJPO JOTUFBEPGHFUUJOHBCTPMVUF
    WBMVFT SFMBUJWFDPNQBSJTPOEBUBBSFVTFE
    ˠ1SFGFSFOUJBM#BZFTJBO0QUJNJ[BUJPO 1#0
    
    ,PZBNBFUBM<>
    Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239.
    4BNQMFEBUBPGDPNQBSJTPOEJ
    'PSUIFDPNQBSJTPOXJUINVMUJQMFPQUJPOT XFBEPQU#SBEMFZ
    5FSSZ-VDF #5-
    NPEFMBTGPMMPXT
    8FJOGFSUIFWBMVFTPGH Y
    CZ."1FTUJNBUJPOVTJOHUIJTNPEFM
    

    View Slide

15. 15
    Preferential Bayesian Optimization in Our Framework
    #FDBVTFH Y
    JTBCMBDLCPYGVODUJPO XFDBOOPUEJSFDUMZTPMWF
    UIJTPQUJNJ[BUJPOQSPCMFN
    *O#BZFTJBOPQUJNJ[BUJPO BDRVJTJUJPOGVODUJPO B JTVTFEUP
    EFUFSNJOFUIFOFYUTBNQMFTJOFBDIJUFSBUJPO
    
    
    
    H Y
    
    

    View Slide

16. 16
    Preferential Bayesian Optimization in Our Framework
    
    
    
    H Y
    
    5IFBDRVJTJUJPOGVODUJPOCBMBODFTFYQMPSBUJPO GBWPSVOFYQMPSFE
    BSFBT
    BOEFYQMPJUBUJPO GBWPSIJHIFYQFDUBUJPOBSFBT
    
    4FBSDIJOHVOFYQMPSFEBSFB
    5IFDPMPSNBQSFQSFTFOUTBDRVJTJUJPOGVODUJPOWBMVFTUIFSFEQPJOU
    SFQSFTFOUTUIFOFYUTBNQMF
    
    

    View Slide

17. *GPOMZUIFVTFSTTFMFDUJPOJOGPSNBUJPOJTVTFE UIFTUBOEBSEBDRVJTJUJPOGVODUJPOJTFOPVHI
    8FJOUSPEVDFBOBEWBODFEBDRVJTJUJPOGVODUJPOGPSVTJOHUIFVTFSTSPVHIQBJOUJOH *JHVJEF
    BTBOBEEJUJPOBMHVJEF
    17
    PBO with Paint-based Guidance in Our Framework
    Selection Paint
    #Iteration 3
    #Iteration 1
    Selection Paint
    a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide) a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide)
    Selection Paint
    #Iteration 2
    a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide)
    

    View Slide

18. 18
    5IJTJTUIFFYUFOEFEWFSTJPOPG<$IPOHFUBM>
    "OFXUFSNGPSUIFHVJEFQBJOUJOH (
    JTBEEFE
    Chong, Toby, et al. "Interactive Optimization of Generative Image Modelling using
    Sequential Subspace Search and Content-based Guidance." Computer Graphics
    Forum. Vol. 40. No. 1. 2021.
    PBO with Paint-based Guidance in Our Framework
    5PPQUJNJ[FGPSUIFOFXBDRVJTJUJPOGVODUJPO XFOFFEUPFWBMVBUF* Y
    
    GPSBSCJUSBSZYIPXFWFS QIZTJDBMMZUBLJOHBQIPUPFBDIUJNF
    UBLFT
    UPPNVDIUJNF BOE
    JTOPUEJGGFSFOUJBCMF
    ˠ-JHIUJOH1SFEJDUJPO OFYUTMJEF
    
    TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH
    DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU
    UIFHSBEJFOUPGUIFBDRVJTJUJPOGVODUJPO
    

    View Slide

19. TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH
    DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU
    19
    8FBEPQU(BVTTJBOQSPDFTTSFHSFTTPS (13
    POFPGTBNQMFCBTFE
    QSFEJDUJPONFUIPET BTBEJGGFSFOUJBCMFMJHIUJOHQSFEJDUJPONFUIPE
    5PUSBJOUIJTSFHSFTTJPONPEFM XFHBUIFSUSBJOJOHEBUBCZBVUPNBUJDBMMZ
    UBLJOHQIPUPTXJUISBOEPNWBSJBCMFTCFGPSFIBOE dNJO
    
    8FDIPTF(13GPSJUTTJNQMJDJUZ
    
    Lighting Prediction in Our Framework
    Ground Truth Prediction
    

    View Slide

20. 20
    Overall Workflow
    $BQUVSJOHUSBJOJOHEBUBGPSMJHIUJOHQSFEJDUJPO
    BOE
    $POTUSVDUJOHSFHSFTTPSGPSMJHIUJOHQSFEJDUJPO
    

    View Slide

21. 21
    Overall Workflow
    $IPPTFJOJUJBMSBOEPNMJHIUJOHDPO
    fi
    HVSBUJPOTGPS
    fi
    STUJUFSBUJPO
    

    View Slide

22. 22
    Overall Workflow
    6TFSTFMFDUJPOBOEQBJOUJOH
    

    View Slide

23. 23
    Overall Workflow
    5IFTZTUFNTFBSDIFTOFYUPQUJPOTVTJOH#BZFTJBOPQUJNJ[BUJPO
    

    View Slide

24. Live Demonstration
    
    
    (simulated virtual environment; not physical environment)
    

    View Slide

25. Result & Evaluation
    

    View Slide

26. 26
    Result
    B
    1PTJUJPO %
    
    C
    3FqFDUPSBOE
    -JHIU1PTJUJPO %
    
    D
    1PTJUJPOBOE
    *OUFOTJUZ %
    
    F
    1IZTJDBM
    3PCPU"SNT %
    
    *UFSBUJPO *UFSBUJPO *UFSBUJPO
    E
    -JHIUT1PTJUJPO
    BOE$PMPS %
    
    4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU
    8FIBWFEFTJHOTFTTJPOTSFTVMUTVOEFSWBSJPVT
    DPOEJUJPOT WBSJPVTJOQVUWBSJBCMFT
    
    'PSFYBNQMF
    w UIFEJNFOTJPOPGUIFJOQVUWBSJBCMFT
    w WBSJPVTMJHIUJOHQBSBNFUFST
    w TJNVMBUJPOPSQIZTJDBMFOWJSPONFOU
    

    View Slide

27. 27
    Technical Evaluation
    Target Paint
    (a) 3D (Light Position) (b) 5D (Light and Reflector Position)
    Target Paint
    (c) 8D (2 Lights Position and Power)
    Target Paint
    (d) 12D (2 Lights Position and Color)
    Target Paint
    

    View Slide

28. 28
    User Study
    (a) Proposed Framework
    Paint Interface
    White Light
    with Robot Arm
    Orange Light
    with Robot Arm
    Reflector
    with Robot Arm
    White Light
    with Tripod
    Orange Light
    with Tripod
    Reflector
    with Tripod
    (b) Traditional Framework
    Captured image on
    camera is displayed
    Shooting Objects
    Shooting Objects
    8FDPOEVDUFEUIFVTFSTUVEZXJUIQBSUJDJQBOUT
    8FQSFQBSFEUZQFTPGUIFGSBNFXPSLGPSUIFQIPUPHSBQIJDMJHIUJOHEFTJHO USBEJUJPOBM
    NBOVBMBEKVTUNFOUBOEPVSGSBNFXPSL
    
    

    View Slide

29. 29
    User Study
    0VST .BOVBM
    (a) Questionnaires
    (b) Satisfaction
    Q Q
    
    *OBMMRVFTUJPOT PVSQSPQPTFEGSBNFXPSLXBTFWBMVBUFECFUUFSUIBOUIF
    USBEJUJPOBMGSBNFXPSL
    &TQFDJBMMZ 2JTUIFNPTUJNQPSUBOURVFTUJPOGPSUIFWBMJEBUJPOPGPVS
    DPOUSJCVUJPOT
    *UJTJOEJDBUFEUIBUQFPQMFDBOCFDPNFGPDVTJOHPOUIFEFTJHOJOHUBTL
    

    View Slide

30. 30
    Discussion
    -JNJUBUJPOTPGPVSVTFSTUVEZ
    )JHIFSEJNFOTJPOBMJUZ
    #0JTOPUHPPEBUIJHIEJNFOTJPOBMQSPCMFNT
    "DDVSBDZPG-JHIUJOH1SFEJDUJPO
    *UTFOPVHIGPSSPVHIHVJEJOH CVUOPUWFSZBDDVSBUF
    *OUFHSBUJPOXJUIFYQFSULOPXMFEHFBOEQBUUFSOT
    *ODPSQPSBUJOHNPSFQBSBNFUFSTGPSQIPUPHSBQIZ
    &YUFOEJOHGFBTJCMFMJHIUJOHEFTJHOT
    &TQFDJBMMZ QIZTJDBMTFUVQJTIFBWJMZSFTUSJDUFECZIBSEXBSFTFUVQ
    5PXBSETWJSUVBMQSPEVDUJPOBOEDJOFNBUPHSBQIZ
    

    View Slide

31. Photographic Lighting Design with
    
    
    Photographer-in-the-Loop Bayesian Optimization
    Kenta Yamamoto1 / Yuki Koyama2 / Yoichi Ochiai1
    1 University of Tsukuba, Digital Nature Group
    
    
    2 National Institute of Advanced Industrial Science and Technology (AIST)
    

    View Slide