Learning with Fenchel-Young Losses

617139a654928632d4e1a9d43fbd2e2c?s=47 Han Bao
June 10, 2020
Science
0
110

Learning with Fenchel-Young Losses

I read the paper "Learning with Fenchel-Young Losses" (JMLR2020): https://arxiv.org/abs/1901.02324

617139a654928632d4e1a9d43fbd2e2c?s=128

Han Bao

June 10, 2020
Tweet

Other Decks in Science

See All in Science
7feb7bbc3605d995c6099de0e25b4b99?s=48 dwhgg
0
430
A908657344fb7d0f37cdc18ae5148a5b?s=48 kojiro0208
0
2.5k
A2cac4b3dcb2bc0b87917ddc034ef708?s=48 sansandsoc
0
2.2k
D8d463da5ab4a99265ffc1d12e76cbc9?s=48 sagara
0
380
38e2af7f8bdad4f2087ab3d42b627e33?s=48 shuntaros
0
120
0ece01ad1513ae4ec0cdb062f9c3e6d8?s=48 pacocat
0
170
Cef25a38df4639275be8726a6d5f1906?s=48 shiftpn
1
260
0b26590a0a8b0f1da140ed5de9b68379?s=48 usamik26
0
410
097c045028b149ca7fd4ca42ff859cd9?s=48 ko_fujita1
1
1.4k
4d5544d0ea612705b35dcca4922e3864?s=48 yohrn
0
340
38e2af7f8bdad4f2087ab3d42b627e33?s=48 shuntaros
1
270
38e2af7f8bdad4f2087ab3d42b627e33?s=48 shuntaros
0
140

Featured

See All Featured
Aff5641764408271f7bc398f2097edd0?s=48 denniskardys
220
110k
9128d500301ae51524e887bb680f471d?s=48 caitiem20
306
17k
053e75a5b48b44d6dd0612795dfb326d?s=48 notwaldorf
3
1.2k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
71
3.5k
C0b42577cfc2b321be3474618107e933?s=48 samanthasiow
56
6.2k
D67fff74178013024d833b3eec6cf27f?s=48 lynnandtonic
268
16k
D09fad3e36ae6841f54820be0e907f7a?s=48 rocio
155
11k
C35e01544a0dd94a2cf1619ee8a42ebb?s=48 clairelew
14
1.4k
E76911cbe088e5b850d966de3fc7435b?s=48 robhawkes
52
2.7k
44b54d2ffcb7857004071cc6795dde11?s=48 cassininazir
346
20k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
39
2.2k
651dcfe41723207fbb0aa044a4f7c07f?s=48 dotmariusz
94
5.1k

Transcript

  1. -FBSOJOHXJUI 'FODIFM:PVOH-PTTFT $SFBUFECZ)BO#BP 1I%BU65PLZP$4 <#MPOEFM .BSUJOTBOE/JDVMBF+.-3>

  2. 8IBUJTMPTTGVODUJPOT ˙ .FBTVSJOHEJ⒎FSFODFCFUXFFOUBSHFUBOEQSFEJDUJPO ⾣ &YBNQMFSFHSFTTJPO ⾣ &YBNQMFCJOBSZDMBTTJpDBUJPO   

    yf(x) ℓ(yf(x)) DPSSFDU XSPOH  y − f(x) ℓ(y − f(x)) NBLJOH DMPTFSUP  TRVBSFEMPTT )VCFSMPTT f(x) y NBLJOH FRVBMUP  MPTT MPHJTUJDMPTT IJOHFMPTT sign( f(x)) sign(y)

  3. 'FODIFM:PVOH-PTT  %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS  Ω : ℝd → ℝ

    LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF ∈ ℝd UBSHFUMBCFM ∈ dom(Ω) 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 8IBUPOUIFFBSUIEPFTJUNFBO  1PUFOUJBMRVFTUJPOT UPCFBOTXFSFE  28IBUJTlQSFEJDUJPOzSFHVMBSJ[FS  28IZEPXFOFFESFHVMBSJ[BUJPOPGQSFEJDUJPO  28IZJTUIFMPTTEFpOFEBTBCPWF

  4. 1JQFMJOFPG4VQFSWJTFE-FBSOJOH  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF ℝd x θ ̂ y

    QBSBNFUSJ[FENPEFM fW QSFEJDUJPOGVODUJPO ̂ yΩ 0.821 1.215 ⋮ 5.382 ⋮ −1.012 0 0 ⋮ 1 ⋮ 0 ∈ %// fW BSHNBY ̂ yΩ *OQVU 4DPSF 0VUQVU &YBNQMF DMBTTJpDBUJPO IPUWFDT

  5. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩

  6. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3  

     BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF

  7. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3  

     BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF OPOVOJRTPMVUJPO Δ3    BSHNBY

  8. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3  

     BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF OPOVOJRTPMVUJPO Δ3    BSHNBY OPOEJ⒎FSFOUJBCMF OPVODFSUBJOUZ

  9. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ

  10.    TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩

    + HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ

  11.    TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩

    + HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ EJ⒎FSFOUJBCMF VODFSUBJOUZ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ

  12.    TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩

    + HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ EFOTFTVQQPSU  GPSMBSHF  JTJOUSBDUBCMF ∑d j=1 d EJ⒎FSFOUJBCMF VODFSUBJOUZ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ

  13. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ

  14. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF

  15. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2

  16. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2 EDBTF Δ3 EFOTF TQBSTF

  17. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2 EDBTF Δ3 EFOTF TQBSTF ⾣ QPJOUTJO ˠEFOTFQSPK ⾣ PUIFSXJTFˠTQBSTFQSPK ⾣ JTGBSTNBMMFSUIBOℝd

  18. 1SFEJDUJPO'VODUJPOT  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ TQBSTFNBY USBDUBCMF  TUJMMJUEFQFOET  EJ⒎FSFOUJBCMF VOJRVFTPMVUJPO TQBSTFTVQQPSU ˠJOUFSQSFUBCMF ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2

  19. l3FHVMBSJ[FEz1SFEJDUJPO  BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ %FpOJUJPO-FU CFBSFHVMBSJ[FS 5IFQSFEJDUJPOGVODUJPOSFHVMBSJ[FECZ JT  Ω : ℝd → ℝ Ω ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) QSFEJDUJPO TDPSF ∈ ℝd NBLFTQSFEJDUJPO BQBSUGSPNWFSUJDFT CFBXBSFEJ⒎FSFOU GSPNVTVBMSFHVMBSJ[BUJPO Loss( fW ) + λ∥W∥2 F

  20. 'VSUIFS4USVDUVSFE1SFEJDUJPO  UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF4FRVFODFMBCFMJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT * MPWF MPTT GVODUJPOT /

    / / / 7 / / / 7 / / / ʜ ʜ JOQVUx PVUQVU DBOETy    ʜ ʜ TDPSFTθ MFOHUIn 7 / + ʜ TJ[Fm TFUPGMBCFMT QSPCBCJMJUZ TJNQMFY

  21. 'VSUIFS4USVDUVSFE1SFEJDUJPO  UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF4FRVFODFMBCFMJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT * MPWF MPTT GVODUJPOT /

    / / / 7 / / / 7 / / / ʜ ʜ JOQVUx PVUQVU DBOETy    ʜ ʜ TDPSFTθ MFOHUIn 7 / + ʜ TJ[Fm TFUPGMBCFMT  FYQPOFOUJBM || = mn QSPCBCJMJUZ TJNQMFY

  22. 'VSUIFS4USVDUVSFE1SFEJDUJPO  UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF-JOFBSBTTJHONFOU FHMJTUXJTFSBOLJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT JOQVUx PVUQVU DBOETy 

      ʜ ʜ TDPSFTθ         #JSLIP⒎ QPMZUPQF EPD  EPD  EPD  EPD  PGEPDTn     ʜ ʜ

  23. 'VSUIFS4USVDUVSFE1SFEJDUJPO  UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF-JOFBSBTTJHONFOU FHMJTUXJTFSBOLJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT JOQVUx PVUQVU DBOETy 

      ʜ ʜ TDPSFTθ         #JSLIP⒎ QPMZUPQF EPD  EPD  EPD  EPD  PGEPDTn  FYQPOFOUJBM || = n!     ʜ ʜ

  24. 'VSUIFS4USVDUVSFE1SFEJDUJPO ˙ -PXEJNFOTJPOBMJOIFSFOUTUSVDUVSFFYJTUT UIP JTFYQPOFOUJBMMZMBSHF     

     ˙ &YBNQMF4FRVFODFMBCFMJOH ⾣ "TTVNQJOQVUXPSENBUUFST ⾣ "TTVNQQSFWMBCFMNBUUFST ||  UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU x1 x2 x3 x4 y1 y2 y3 y4   *OQVUTQBDF 4DPSFTQBDF x θ NPEFM fW -PXEJN TDPSFTQBDF η MJOFBSUSBOT M QSPCMFNEFQFOEFOU 㱺MPXEJNTUSVDUVSFO(nm2) XJEFMZVTFEJOMJOFBSDIBJO$3'T

  25. 'VSUIFS4USVDUVSFE1SFEJDUJPO  ."1JOGFSFODF argmax y∈conv() ⟨θ, y⟩ NBSHJOBMJOGFSFODF argmax y∈conv()

    ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ 4QBSTF."1 argmax y∈conv() ⟨θ, y⟩ + H2 (y) 5TBMMJT FOUSPQZ UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU USBDUBCMF OPVODFSUBJOUZ OPEJ⒎FSFOUJBUJPO EJ⒎FSFOUJBCMF VODFSUBJOUZ  PGUFO JOUSBDUBCMF EFOTFTVQQPSU EJ⒎FSFOUJBCMF VODFSUBJOUZ USBDUBCMF 'SBOL8PMGF  TQBSTFTVQQPSU 3FNBSL TQBSTFNBYEPFTOPUVUJMJ[F MPXEJNTUSVDUVSF 3FNBSL5SBDUBCJMJUZ TFRVFODFMBCFMJOH."1 7JUFSCJ NBSHJOBMBSF  MJOBTTJHO."1 )VOHBSJBO JT NBSHJOBMJT1DPNQ O(nm2) O(n3)

  26. )PXUPEFTJHOMPTT  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2

  27. )PXUPEFTJHOMPTT  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 UBSHFUMBCFM y

  28. )PXUPEFTJHOMPTT  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 UBSHFUMBCFM y 2)PXUPNFBTVSF

  29. )PXUPEFTJHOMPTT  TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)

    TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO

  30. )PXUPEFTJHOMPTT  TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)

    TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT

  31. )PXUPEFTJHOMPTT  TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)

    TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT 28IZJUJTHPPE

  32. )PXUPEFTJHOMPTT  TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)

    TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT ʁʁʁ 28IZJUJTHPPE

  33. )PXUPEFTJHOMPTT  TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)

    TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT ʁʁʁ 28IZJUJTHPPE 2)PXUPEFTJHO

  34. 'FODIFM:PVOH-PTT  %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS  Ω : ℝd → ℝ

    LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙  DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ)

  35. 'FODIFM:PVOH-PTT  %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS  Ω : ℝd → ℝ

    LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙  DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ) .JOJNJ[JOH':MPTTNBLFTQSFEJDUJPODMPTFUPUBSHFUMBCFM

  36. 'FODIFM:PVOH-PTT  %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS  Ω : ℝd → ℝ

    LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙  DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ) .JOJNJ[JOH':MPTTNBLFTQSFEJDUJPODMPTFUPUBSHFUMBCFM 1SPPG6TF'FODIFM:PVOHJOFRVBMJUZ Ω⋆(θ) + Ω(y) ≥ {⟨θ, y⟩ − Ω(y)} + Ω(y) = ⟨θ, y⟩

  37. (FPNFUSJDBM*OUFSQSFUBUJPO  Ω(y) y μ 'PSSFHVMBSJ[FS

  38. (FPNFUSJDBM*OUFSQSFUBUJPO  Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU

    BU    CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ)

  39. (FPNFUSJDBM*OUFSQSFUBUJPO  Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU

    BU    CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ) ⟨θ, y⟩ − Ω⋆(θ) −Ω⋆(θ)

  40. (FPNFUSJDBM*OUFSQSFUBUJPO  Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU

    BU    CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ) ⟨θ, y⟩ − Ω⋆(θ) −Ω⋆(θ) LΩ (y; θ) -PTTJTEJTUBODFCFUXFFO BOE BU  #SFHNBOEJWFSHFODF y

  41. &YBNQMF4IBOOPO&OUSPQZ  HS (y) = − d ∑ j=1 yj

    log yj ̂ yHS (θ) = argmax y∈Δd ⟨θ, y⟩ − HS (y) = exp θ ∑d j=1 exp θj TPGUNBY   θ ̂ y(θ) CJOBSZTPGUNBYTJHNPJE LHS (θ; y) = H⋆ S (θ) + HS (y) − ⟨θ, y⟩ = log d ∑ j=1 exp θj − θk BTTVNJOHy = ek DSPTTFOUSPQZ JTMPHJTUJDMPTTJOCJOBSZDBTF LHS ( ̂ yHS (θ); y)

  42. &YBNQMF5TBMMJT&OUSPQZ  H2 (y) = 1 2 d ∑ j=1

    yj (1 − yj ) Hα (y) = 1 α(α − 1) d ∑ j=1 (yj − yα j ) HS (y) = − d ∑ j=1 yj log yj 5TBMMJTFOUSPQZ α α → 2 α → 1 BLB(JOJJOEFY 4IBOOPOFOUSPQZ TQBSTFNBY   θ ̂ y(θ)  m ℓ(m)  TQBSTFNBYMPTT H⋆ 2 (θ) + H2 (y) − ⟨θ, y⟩ NPEJpFE)VCFSMPTT TQFDJBMJ[FEJOCJOBSZDMBTTJpDBUJPO

  43. 0UIFS/JDF1SPQFSUZ 0WFSWJFX ˙ 4FQBSBUJPONBSHJO j j  BpOJUFTDPSFBUUBJOT[FSPMPTT JG JTlTQBSTFz

           ˙ $BMJCSBUFETVSSPHBUF j  NJOJNJ[JOH':MPTTMFBETUPNJOJNJ[JOHDMBTTJpDBUJPOFSSPS NPSFEJTDVTTJPOJTOFFEFEGPSTUSVDUVSFEQSFEJDUJPO  ˙ &⒏DJFOUPQUJNJ[BUJPO j  BMXBZTDPOWFYCZOBUVSFPQUJNJ[BCMFXJUI'SBOL8PMGFBMHPSJUIN JUFSBUJWFMZNJOJNJ[JOHMJOFBSBQQSPY Ω   m ℓ(m) MPHJTUJD 4IBOOPO WBOJTIFTBU 㱣OPTFQNHO TQBSTFNBY 5TBMMJT TFQNHO  m ℓ(m)  㱺OPQFOBMJ[BUJPOPOMBSHFFOPVHIQSFEJDUJPONBSHJOT

  44. 4VNNBSZ  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y UBSHFUMBCFM y

  45. 4VNNBSZ  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y UBSHFUMBCFM y ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) 3FHVMBSJ[FEQSFEJDUJPO NBLFTQBSTF USBDUBCMF ʜ

  46. 4VNNBSZ  *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ

    QSFEJDUJPO ̂ y UBSHFUMBCFM y ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) 3FHVMBSJ[FEQSFEJDUJPO NBLFTQBSTF USBDUBCMF ʜ LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ 'FODIFM:PVOHMPTT TZTUFNBUJDXBZDPOTUSVDUJOHMPTTGSPNΩ