Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Digital Nature Group

January 29, 2023
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. Photographic Lighting Design with


    Photographer-in-the-Loop Bayesian Optimization
    Kenta Yamamoto1 / Yuki Koyama2 / Yoichi Ochiai1
    1 University of Tsukuba, Digital Nature Group


    2 National Institute of Advanced Industrial Science and Technology (AIST)

    View Slide

  2. 2
    Overview of Our Proposal
    5SBEJUJPOBMEFTJHONFUIPE 1SPQPTFEGSBNFXPSL
    w /PWFMEFTJHOGSBNFXPSLXJUIPVUNBOVBMBEKVTUNFOUT
    w *OUVJUJWFEFTJHOPQFSBUJPOJODMVEJOHQBJOUJOHPOTIPUT
    w .BOVBMBEKVTUNFOUTBSFSFBMMZIFBWZQSPDFTT
    w 5SJBMBOEFSSPSQSPDFTTUBLFTNVDIUJNF

    View Slide

  3. 3
    Related Work
    Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and
    interactive techniques. 1993.
    Murmann, Lukas, et al. "Computational bounce flash for indoor
    portraits." ACM Transactions on Graphics (TOG) 35.6 (2016): 1-9.
    Srikanth, Manohar, Kavita Bala, and Frédo Durand.
    "Computational rim illumination with aerial robots." Proceedings
    of the Workshop on Computational Aesthetics. 2014.
    Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing
    Portrait Lighting at Capture-Time Using a 360 Camera as a Light
    Probe." UIST. 2019.
    Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted
    image compositing for photographic lighting." ACM Trans.
    Graph. 32.4 (2013): 36-1.
    JO$(XPSME "VUPNBUJD'MBTI
    1PTUQSPDFTTJOH
    BU$BQUVSFUJNFTVQQPSU
    %SPOFMJHIUJOH

    View Slide

  4. 4
    Lighting Design Interface in CG World
    Schoeneman, Chris, et al. "Painting with light." Proceedings of the 20th annual conference on Computer graphics and interactive techniques.
    1993.
    Pellacini, Fabio, Parag Tole, and Donald P. Greenberg. "A user
    interface for interactive cinematic shadow design." ACM Transactions
    on Graphics (TOG) 21.3 (2002): 563-566.
    1FMMBDJOJFUBM<5P(>
    4DIPFOFNBOFUBM<>
    8FUBSHFUSFBMXPSMEMJHIUJOHGPSUIFQIPUPHSBQIZXJUIWBSJPVTMJHIUJOHEFWJDFT

    View Slide

  5. 5
    Lighting Design in Real World (automatic lighting)
    Murmann, Lukas, et al. "Computational bounce flash for indoor portraits." ACM Transactions on
    Graphics (TOG) 35.6 (2016): 1-9.
    Srikanth, Manohar, Kavita Bala, and Frédo Durand. "Computational rim illumination with aerial
    robots." Proceedings of the Workshop on Computational Aesthetics. 2014.
    .VSNBOOFUBM<5P(>
    4SJLBOUIFUBM<>
    0VSHPBMJTBOJOUFSBDUJWFGSBNFXPSLUIBUSF
    fl
    FDUTUIFQIPUPHSBQIFSTQSFGFSFODFT

    View Slide

  6. 6
    Lighting Design in Real World (at capture-time)
    -&FUBM<6*45>
    Jane, L. E., Ohad Fried, and Maneesh Agrawala. "Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light
    Probe." UIST. 2019.
    8FNBOJQVMBUFWBSJPVTMJHIUJOHEFWJDFTBOEUBLFBIVNBOJOUIFMPPQBQQSPBDIUPPQUJNJ[FGPSUIFQSFGFSFODF

    View Slide

  7. 7
    Lighting Design in Real World (post-processing; re-lighting)
    Boyadzhiev, Ivaylo, Sylvain Paris, and Kavita Bala. "User-assisted image compositing for
    photographic lighting." ACM Trans. Graph. 32.4 (2013): 36-1.
    #PZBE[IJFWFUBM<5P(>
    Sun, Tiancheng, et al. "Single image portrait relighting." ACM Trans. Graph. 38.4 (2019): 79-1.
    4VOFUBM< 5P(>
    8FUBSHFUlDBQUVSFUJNFz XIJDIBMMPXTBSUJGBDUGSFFIJHIFTURVBMJUZSFTVMUT

    View Slide

  8. Our Proposed Framework

    View Slide

  9. 9
    Overview of our framework
    1. Selection 2. Painting 3. Search for Next Options
    Photographer-in-the-Loop
    Photographer Part System Part
    w 1IPUPHSBQIFS1BSU4FMFDUJPOBOE1BJOUJOH
    w 4ZTUFN1BSU4FBSDI/FYU0QUJPOT
    8FEF
    fi
    OFUIJTMPPQQSPDFTTBT
    QIPUPHSBQIFSJOUIFMPPQQSPDFTT

    View Slide

  10. 10
    Human-in-the-loop Design Optimization (Human Evaluator)
    1. Selection 2. Painting 3. Search for Next Options
    Photographer-in-the-Loop
    Photographer Part System Part
    5IJTSFTFBSDIJTPOFPGUIF)VNBOJOUIFMPPQ )*5-
    %FTJHO0QUJNJ[BUJPO
    *O)*5-EFTJHOPQUJNJ[BUJPO IVNBOTBSFFWBMVBUPSTGPSTPMWJOHEFTJHOPQUJNJ[BUJPOQSPCMFNT

    XJUIQFSDFQUVBMPCKFDUJWFGVODUJPO
    &WBMVBUPSXJUIQFSDFQUVBM
    PCKFDUJWFGVODUJPO
    )VNBO

    View Slide

  11. %FTJHOPQUJNJ[BUJPODBOCFEF
    fi
    OFEBTUIFQBSBNFUFS
    PQUJNJ[BUJPO
    'PSFYBNQMF UIFSFJTUIFBSSBOHFNFOUUBTLPGB
    QIPUPHSBQIFOIBODFNFOU
    5IFOVNCFSPGTMJEFSTJTUIFEJNFOTJPOPGUIFJOQVU
    WBSJBCMFT BOEUIFVTFSTFBSDIFTUIFPQUJNBMQBSBNFUFS
    TFUPGUIFN
    *EPOUMJLFUIJTDPMPS
    $IBOHFXIJUFCBMBODFUPCMVF
    11
    Human-in-the-loop Design Optimization (Parameter Optimization)
    FYQIPUPHSBQIFOIBODFNFOU
    )VNBO

    View Slide

  12. 8FEF
    fi
    OFUIFVTFSQSFGFSFODFBTUIFPCKFDUJWFGVODUJPO

    IFSF XFDBMM(PPEOFTT'VODUJPOH Y


    8FXBOUUPTPMWFUIFPQUJNJ[BUJPOQSPCMFN
    12
    Problem Formulation (parameter optimization)
    -JHIUJOHEFTJHODBOBMTPCFEF
    fi
    OFEBTBQBSBNFUFSPQUJNJ[BUJPO
    QSPCMFN
    -JHIUJOHQBSBNFUFST Y DBOCFEF
    fi
    OFEBTGPMMPXT

    BMMQBSBNFUFSTBSFOPSNBMJ[FEXJUIJOd




    H Y

    H Y
    JTBCMBDLCPYGVODUJPO
    H Y
    JTBQFSDFQUVBMPCKFDUJWFGVODUJPO
    8FDBOUPCTFSWFJUTGVODUJPOWBMVFTEJSFDUMZ

    View Slide

  13. 13
    Bayesian Optimization (BO)
    #BZFTJBOPQUJNJ[BUJPO #0
    JTXFMMLOPXOBTUIFTBNQMFFG
    fi
    DJFOUQBSBNFUFSTFBSDINFUIPE
    #0DBOCFBQQMJFEUPUIFCMBDLCPYPCKFDUJWFGVODUJPOIPXFWFS

    JUBTTVNFTUIBUXFDBOPCTFSWFUIFBCTPMVUFWBMVFTPGUIFCMBDLCPYGVODUJPO
    Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239.
    ,PZBNBFUBM<>
    4UBOEBSE#0DBOUCFBQQMJFEEJSFDUMZUPIVNBOJOUIFMPPQ
    EFTJHOPQUJNJ[BUJPO
    )VNBOTBSFOPUHPPEBUBOTXFSJOHBCTPMVUF

    WBMVFTBTTPDJBUFEXJUIJOQVUQBSBNFUFST

    View Slide

  14. 14
    Preferential Bayesian Optimization (PBO) / Preferential Data Model
    'PSUIF)*5-EFTJHOPQUJNJ[BUJPO JOTUFBEPGHFUUJOHBCTPMVUF
    WBMVFT SFMBUJWFDPNQBSJTPOEBUBBSFVTFE
    ˠ1SFGFSFOUJBM#BZFTJBO0QUJNJ[BUJPO 1#0

    ,PZBNBFUBM<>
    Koyama, Yuki, Toby Chong, and Takeo Igarashi. "Preferential Bayesian Optimisation for Visual Design." Bayesian Methods for Interaction and Design (2022): 239.
    4BNQMFEBUBPGDPNQBSJTPOEJ
    'PSUIFDPNQBSJTPOXJUINVMUJQMFPQUJPOT XFBEPQU#SBEMFZ
    5FSSZ-VDF #5-
    NPEFMBTGPMMPXT
    8FJOGFSUIFWBMVFTPGH Y
    CZ."1FTUJNBUJPOVTJOHUIJTNPEFM

    View Slide

  15. 15
    Preferential Bayesian Optimization in Our Framework
    #FDBVTFH Y
    JTBCMBDLCPYGVODUJPO XFDBOOPUEJSFDUMZTPMWF
    UIJTPQUJNJ[BUJPOQSPCMFN
    *O#BZFTJBOPQUJNJ[BUJPO BDRVJTJUJPOGVODUJPO B JTVTFEUP
    EFUFSNJOFUIFOFYUTBNQMFTJOFBDIJUFSBUJPO



    H Y

    View Slide

  16. 16
    Preferential Bayesian Optimization in Our Framework



    H Y

    5IFBDRVJTJUJPOGVODUJPOCBMBODFTFYQMPSBUJPO GBWPSVOFYQMPSFE
    BSFBT
    BOEFYQMPJUBUJPO GBWPSIJHIFYQFDUBUJPOBSFBT

    4FBSDIJOHVOFYQMPSFEBSFB
    5IFDPMPSNBQSFQSFTFOUTBDRVJTJUJPOGVODUJPOWBMVFTUIFSFEQPJOU
    SFQSFTFOUTUIFOFYUTBNQMF

    View Slide

  17. *GPOMZUIFVTFSTTFMFDUJPOJOGPSNBUJPOJTVTFE UIFTUBOEBSEBDRVJTJUJPOGVODUJPOJTFOPVHI
    8FJOUSPEVDFBOBEWBODFEBDRVJTJUJPOGVODUJPOGPSVTJOHUIFVTFSTSPVHIQBJOUJOH *JHVJEF
    BTBOBEEJUJPOBMHVJEF
    17
    PBO with Paint-based Guidance in Our Framework
    Selection Paint
    #Iteration 3
    #Iteration 1
    Selection Paint
    a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide) a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide)
    Selection Paint
    #Iteration 2
    a
    i
    (x) G(I(x), I
    i
    guide) a
    i
    guide(x;I
    i
    guide)

    View Slide

  18. 18
    5IJTJTUIFFYUFOEFEWFSTJPOPG<$IPOHFUBM>
    "OFXUFSNGPSUIFHVJEFQBJOUJOH (
    JTBEEFE
    Chong, Toby, et al. "Interactive Optimization of Generative Image Modelling using
    Sequential Subspace Search and Content-based Guidance." Computer Graphics
    Forum. Vol. 40. No. 1. 2021.
    PBO with Paint-based Guidance in Our Framework
    5PPQUJNJ[FGPSUIFOFXBDRVJTJUJPOGVODUJPO XFOFFEUPFWBMVBUF* Y

    GPSBSCJUSBSZYIPXFWFS QIZTJDBMMZUBLJOHBQIPUPFBDIUJNF
    UBLFT
    UPPNVDIUJNF BOE
    JTOPUEJGGFSFOUJBCMF
    ˠ-JHIUJOH1SFEJDUJPO OFYUTMJEF

    TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH
    DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU
    UIFHSBEJFOUPGUIFBDRVJTJUJPOGVODUJPO

    View Slide

  19. TUBOEBSEBDRVJTJUJPOGVODUJPO GPSUIFHVJEFQBJOUJOH
    DPNQBSJTPOCFUXFFOQBJOUJOHBOEMJHIUJOHQSFEJDUJPOSFTVMU
    19
    8FBEPQU(BVTTJBOQSPDFTTSFHSFTTPS (13
    POFPGTBNQMFCBTFE
    QSFEJDUJPONFUIPET BTBEJGGFSFOUJBCMFMJHIUJOHQSFEJDUJPONFUIPE
    5PUSBJOUIJTSFHSFTTJPONPEFM XFHBUIFSUSBJOJOHEBUBCZBVUPNBUJDBMMZ
    UBLJOHQIPUPTXJUISBOEPNWBSJBCMFTCFGPSFIBOE dNJO

    8FDIPTF(13GPSJUTTJNQMJDJUZ

    Lighting Prediction in Our Framework
    Ground Truth Prediction

    View Slide

  20. 20
    Overall Workflow
    $BQUVSJOHUSBJOJOHEBUBGPSMJHIUJOHQSFEJDUJPO
    BOE
    $POTUSVDUJOHSFHSFTTPSGPSMJHIUJOHQSFEJDUJPO

    View Slide

  21. 21
    Overall Workflow
    $IPPTFJOJUJBMSBOEPNMJHIUJOHDPO
    fi
    HVSBUJPOTGPS
    fi
    STUJUFSBUJPO

    View Slide

  22. 22
    Overall Workflow
    6TFSTFMFDUJPOBOEQBJOUJOH

    View Slide

  23. 23
    Overall Workflow
    5IFTZTUFNTFBSDIFTOFYUPQUJPOTVTJOH#BZFTJBOPQUJNJ[BUJPO

    View Slide

  24. Live Demonstration


    (simulated virtual environment; not physical environment)

    View Slide

  25. Result & Evaluation

    View Slide

  26. 26
    Result
    B
    1PTJUJPO %

    C
    3FqFDUPSBOE
    -JHIU1PTJUJPO %

    D
    1PTJUJPOBOE
    *OUFOTJUZ %

    F
    1IZTJDBM
    3PCPU"SNT %

    *UFSBUJPO *UFSBUJPO *UFSBUJPO
    E
    -JHIUT1PTJUJPO
    BOE$PMPS %

    4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU 4FMFDUJPO 1BJOU
    8FIBWFEFTJHOTFTTJPOTSFTVMUTVOEFSWBSJPVT
    DPOEJUJPOT WBSJPVTJOQVUWBSJBCMFT

    'PSFYBNQMF
    w UIFEJNFOTJPOPGUIFJOQVUWBSJBCMFT
    w WBSJPVTMJHIUJOHQBSBNFUFST
    w TJNVMBUJPOPSQIZTJDBMFOWJSPONFOU

    View Slide

  27. 27
    Technical Evaluation
    Target Paint
    (a) 3D (Light Position) (b) 5D (Light and Reflector Position)
    Target Paint
    (c) 8D (2 Lights Position and Power)
    Target Paint
    (d) 12D (2 Lights Position and Color)
    Target Paint

    View Slide

  28. 28
    User Study
    (a) Proposed Framework
    Paint Interface
    White Light
    with Robot Arm
    Orange Light
    with Robot Arm
    Reflector
    with Robot Arm
    White Light
    with Tripod
    Orange Light
    with Tripod
    Reflector
    with Tripod
    (b) Traditional Framework
    Captured image on
    camera is displayed
    Shooting Objects
    Shooting Objects
    8FDPOEVDUFEUIFVTFSTUVEZXJUIQBSUJDJQBOUT
    8FQSFQBSFEUZQFTPGUIFGSBNFXPSLGPSUIFQIPUPHSBQIJDMJHIUJOHEFTJHO USBEJUJPOBM
    NBOVBMBEKVTUNFOUBOEPVSGSBNFXPSL

    View Slide

  29. 29
    User Study
    0VST .BOVBM
    (a) Questionnaires
    (b) Satisfaction
    Q Q

    *OBMMRVFTUJPOT PVSQSPQPTFEGSBNFXPSLXBTFWBMVBUFECFUUFSUIBOUIF
    USBEJUJPOBMGSBNFXPSL
    &TQFDJBMMZ 2JTUIFNPTUJNQPSUBOURVFTUJPOGPSUIFWBMJEBUJPOPGPVS
    DPOUSJCVUJPOT
    *UJTJOEJDBUFEUIBUQFPQMFDBOCFDPNFGPDVTJOHPOUIFEFTJHOJOHUBTL

    View Slide

  30. 30
    Discussion
    -JNJUBUJPOTPGPVSVTFSTUVEZ
    )JHIFSEJNFOTJPOBMJUZ
    #0JTOPUHPPEBUIJHIEJNFOTJPOBMQSPCMFNT
    "DDVSBDZPG-JHIUJOH1SFEJDUJPO
    *UTFOPVHIGPSSPVHIHVJEJOH CVUOPUWFSZBDDVSBUF
    *OUFHSBUJPOXJUIFYQFSULOPXMFEHFBOEQBUUFSOT
    *ODPSQPSBUJOHNPSFQBSBNFUFSTGPSQIPUPHSBQIZ
    &YUFOEJOHGFBTJCMFMJHIUJOHEFTJHOT
    &TQFDJBMMZ QIZTJDBMTFUVQJTIFBWJMZSFTUSJDUFECZIBSEXBSFTFUVQ
    5PXBSETWJSUVBMQSPEVDUJPOBOEDJOFNBUPHSBQIZ

    View Slide

  31. Photographic Lighting Design with


    Photographer-in-the-Loop Bayesian Optimization
    Kenta Yamamoto1 / Yuki Koyama2 / Yoichi Ochiai1
    1 University of Tsukuba, Digital Nature Group


    2 National Institute of Advanced Industrial Science and Technology (AIST)

    View Slide