Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning with Fenchel-Young Losses
Search
Han Bao
June 10, 2020
Science
0
340
Learning with Fenchel-Young Losses
I read the paper "Learning with Fenchel-Young Losses" (JMLR2020):
https://arxiv.org/abs/1901.02324
Han Bao
June 10, 2020
Tweet
Share
Other Decks in Science
See All in Science
白金鉱業Meetup Vol.16_数理最適化案件のはじめかた・すすめかた
brainpadpr
3
1.8k
CV_3_Keypoints
hachama
0
190
科学で迫る勝敗の法則(名城大学公開講座.2024年10月) / The principle of victory discovered by science (Open lecture in Meijo Univ. 2024)
konakalab
0
350
実力評価性能を考慮した弓道高校生全国大会の大会制度設計の提案 / (konakalab presentation at MSS 2025.03)
konakalab
2
170
統計学入門講座 第2回スライド
techmathproject
0
130
[第62回 CV勉強会@関東] Long-CLIP: Unlocking the Long-Text Capability of CLIP / kantoCV 62th ECCV 2024
lychee1223
1
940
データベース11: 正規化(1/2) - 望ましくない関係スキーマ
trycycle
PRO
0
640
点群ライブラリPDALをGoogleColabにて実行する方法の紹介
kentaitakura
1
290
Valuable Lessons Learned on Kaggle’s ARC AGI LLM Challenge (PyDataGlobal 2024)
ianozsvald
0
390
統計学入門講座 第4回スライド
techmathproject
0
140
機械学習 - SVM
trycycle
PRO
1
820
データベース08: 実体関連モデルとは?
trycycle
PRO
0
670
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Making the Leap to Tech Lead
cromwellryan
134
9.3k
Navigating Team Friction
lara
187
15k
Raft: Consensus for Rubyists
vanstee
140
7k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Side Projects
sachag
455
42k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Transcript
-FBSOJOHXJUI 'FODIFM:PVOH-PTTFT $SFBUFECZ)BO#BP 1I%BU65PLZP$4 <#MPOEFM .BSUJOTBOE/JDVMBF+.-3>
8IBUJTMPTTGVODUJPOT ˙ .FBTVSJOHEJ⒎FSFODFCFUXFFOUBSHFUBOEQSFEJDUJPO ⾣ &YBNQMFSFHSFTTJPO ⾣ &YBNQMFCJOBSZDMBTTJpDBUJPO
yf(x) ℓ(yf(x)) DPSSFDU XSPOH y − f(x) ℓ(y − f(x)) NBLJOH DMPTFSUP TRVBSFEMPTT )VCFSMPTT f(x) y NBLJOH FRVBMUP MPTT MPHJTUJDMPTT IJOHFMPTT sign( f(x)) sign(y)
'FODIFM:PVOH-PTT %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS Ω : ℝd → ℝ
LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF ∈ ℝd UBSHFUMBCFM ∈ dom(Ω) 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 8IBUPOUIFFBSUIEPFTJUNFBO 1PUFOUJBMRVFTUJPOT UPCFBOTXFSFE 28IBUJTlQSFEJDUJPOzSFHVMBSJ[FS 28IZEPXFOFFESFHVMBSJ[BUJPOPGQSFEJDUJPO 28IZJTUIFMPTTEFpOFEBTBCPWF
1JQFMJOFPG4VQFSWJTFE-FBSOJOH *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF ℝd x θ ̂ y
QBSBNFUSJ[FENPEFM fW QSFEJDUJPOGVODUJPO ̂ yΩ 0.821 1.215 ⋮ 5.382 ⋮ −1.012 0 0 ⋮ 1 ⋮ 0 ∈ %// fW BSHNBY ̂ yΩ *OQVU 4DPSF 0VUQVU &YBNQMF DMBTTJpDBUJPO IPUWFDT
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3
BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3
BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF OPOVOJRTPMVUJPO Δ3 BSHNBY
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ Δ3
BSHNBY ˙ DIPPTJOHBWFSUFY USBDUBCMF OPOVOJRTPMVUJPO Δ3 BSHNBY OPOEJ⒎FSFOUJBCMF OPVODFSUBJOUZ
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ
TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩
+ HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ
TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩
+ HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ EJ⒎FSFOUJBCMF VODFSUBJOUZ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ
TPGUNBY ˙ SFHVMBSJ[FUPXBSETDFOUFS argmax y∈Δd ⟨θ, y⟩
+ HS (y) = exp θi ∑d j=1 exp θj i PSEJOBSZFYQSFTTJPO 1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ EFOTFTVQQPSU GPSMBSHF JTJOUSBDUBCMF ∑d j=1 d EJ⒎FSFOUJBCMF VODFSUBJOUZ TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2 EDBTF Δ3 EFOTF TQBSTF
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2 UFOEUPCFTQBSTF EDBTF EFOTF TQBSTF Δ2 EDBTF Δ3 EFOTF TQBSTF ⾣ QPJOUTJO ˠEFOTFQSPK ⾣ PUIFSXJTFˠTQBSTFQSPK ⾣ JTGBSTNBMMFSUIBOℝd
1SFEJDUJPO'VODUJPOT BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ TQBSTFNBY USBDUBCMF TUJMMJUEFQFOET EJ⒎FSFOUJBCMF VOJRVFTPMVUJPO TQBSTFTVQQPSU ˠJOUFSQSFUBCMF ˙ &VDMJEFBOQSPKFDUJPOUPXBSETTJNQMFY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 = argmax y∈Δd ∥y − θ∥2
l3FHVMBSJ[FEz1SFEJDUJPO BSHNBY argmax y∈Δd ⟨θ, y⟩ TPGUNBY argmax y∈Δd
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 5TBMMJT FOUSPQZ %FpOJUJPO-FU CFBSFHVMBSJ[FS 5IFQSFEJDUJPOGVODUJPOSFHVMBSJ[FECZ JT Ω : ℝd → ℝ Ω ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) QSFEJDUJPO TDPSF ∈ ℝd NBLFTQSFEJDUJPO BQBSUGSPNWFSUJDFT CFBXBSFEJ⒎FSFOU GSPNVTVBMSFHVMBSJ[BUJPO Loss( fW ) + λ∥W∥2 F
'VSUIFS4USVDUVSFE1SFEJDUJPO UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF4FRVFODFMBCFMJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT * MPWF MPTT GVODUJPOT /
/ / / 7 / / / 7 / / / ʜ ʜ JOQVUx PVUQVU DBOETy ʜ ʜ TDPSFTθ MFOHUIn 7 / + ʜ TJ[Fm TFUPGMBCFMT QSPCBCJMJUZ TJNQMFY
'VSUIFS4USVDUVSFE1SFEJDUJPO UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF4FRVFODFMBCFMJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT * MPWF MPTT GVODUJPOT /
/ / / 7 / / / 7 / / / ʜ ʜ JOQVUx PVUQVU DBOETy ʜ ʜ TDPSFTθ MFOHUIn 7 / + ʜ TJ[Fm TFUPGMBCFMT FYQPOFOUJBM || = mn QSPCBCJMJUZ TJNQMFY
'VSUIFS4USVDUVSFE1SFEJDUJPO UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF-JOFBSBTTJHONFOU FHMJTUXJTFSBOLJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT JOQVUx PVUQVU DBOETy
ʜ ʜ TDPSFTθ #JSLIP⒎ QPMZUPQF EPD EPD EPD EPD PGEPDTn ʜ ʜ
'VSUIFS4USVDUVSFE1SFEJDUJPO UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU &YBNQMF-JOFBSBTTJHONFOU FHMJTUXJTFSBOLJOH 0VUQVUTQBDFDPOTJTUTPGTUSVDUVSFEPCKFDUTTVDIBTHSBQIT JOQVUx PVUQVU DBOETy
ʜ ʜ TDPSFTθ #JSLIP⒎ QPMZUPQF EPD EPD EPD EPD PGEPDTn FYQPOFOUJBM || = n! ʜ ʜ
'VSUIFS4USVDUVSFE1SFEJDUJPO ˙ -PXEJNFOTJPOBMJOIFSFOUTUSVDUVSFFYJTUT UIP JTFYQPOFOUJBMMZMBSHF
˙ &YBNQMF4FRVFODFMBCFMJOH ⾣ "TTVNQJOQVUXPSENBUUFST ⾣ "TTVNQQSFWMBCFMNBUUFST || UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU x1 x2 x3 x4 y1 y2 y3 y4 *OQVUTQBDF 4DPSFTQBDF x θ NPEFM fW -PXEJN TDPSFTQBDF η MJOFBSUSBOT M QSPCMFNEFQFOEFOU 㱺MPXEJNTUSVDUVSFO(nm2) XJEFMZVTFEJOMJOFBSDIBJO$3'T
'VSUIFS4USVDUVSFE1SFEJDUJPO ."1JOGFSFODF argmax y∈conv() ⟨θ, y⟩ NBSHJOBMJOGFSFODF argmax y∈conv()
⟨θ, y⟩ + HS (y) 4IBOOPO FOUSPQZ 4QBSTF."1 argmax y∈conv() ⟨θ, y⟩ + H2 (y) 5TBMMJT FOUSPQZ UIJTQBSUKVTUNPUJWBUFTSFHVMBSJ[BUJPOGVSUIFSZPVNBZTLJQJU USBDUBCMF OPVODFSUBJOUZ OPEJ⒎FSFOUJBUJPO EJ⒎FSFOUJBCMF VODFSUBJOUZ PGUFO JOUSBDUBCMF EFOTFTVQQPSU EJ⒎FSFOUJBCMF VODFSUBJOUZ USBDUBCMF 'SBOL8PMGF TQBSTFTVQQPSU 3FNBSL TQBSTFNBYEPFTOPUVUJMJ[F MPXEJNTUSVDUVSF 3FNBSL5SBDUBCJMJUZ TFRVFODFMBCFMJOH."1 7JUFSCJ NBSHJOBMBSF MJOBTTJHO."1 )VOHBSJBO JT NBSHJOBMJT1DPNQ O(nm2) O(n3)
)PXUPEFTJHOMPTT *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2
)PXUPEFTJHOMPTT *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 UBSHFUMBCFM y
)PXUPEFTJHOMPTT *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y) TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 UBSHFUMBCFM y 2)PXUPNFBTVSF
)PXUPEFTJHOMPTT TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)
TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO
)PXUPEFTJHOMPTT TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)
TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT
)PXUPEFTJHOMPTT TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)
TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT 28IZJUJTHPPE
)PXUPEFTJHOMPTT TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)
TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT ʁʁʁ 28IZJUJTHPPE
)PXUPEFTJHOMPTT TPGUNBY argmax y∈Δd ⟨θ, y⟩ + HS (y)
TQBSTFNBY argmax y∈Δd ⟨θ, y⟩− 1 2 ∥y∥2 2 1SFEJDUJPOGVODUJPO -PTTGVODUJPO DSPTTFOUSPQZ log∑ i exp θi − θk UBSHFUDMBTT ʁʁʁ 28IZJUJTHPPE 2)PXUPEFTJHO
'FODIFM:PVOH-PTT %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS Ω : ℝd → ℝ
LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙ DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ)
'FODIFM:PVOH-PTT %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS Ω : ℝd → ℝ
LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙ DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ) .JOJNJ[JOH':MPTTNBLFTQSFEJDUJPODMPTFUPUBSHFUMBCFM
'FODIFM:PVOH-PTT %FpOJUJPO-FU CFBlQSFEJDUJPOzSFHVMBSJ[FS Ω : ℝd → ℝ
LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ QSFEJDUJPO TDPSF UBSHFUMBCFM 'FODIFMDPOKVHBUF Ω⋆(θ) := sup μ∈dom(Ω) ⟨θ, μ⟩ − Ω(μ) 5XPLFZQSPQFSUJFT ˙ ':MPTTJTOPOOFHBUJWF ˙ DPSSFDUQSFE J⒎[FSPMPTT y = ̂ yΩ (θ) .JOJNJ[JOH':MPTTNBLFTQSFEJDUJPODMPTFUPUBSHFUMBCFM 1SPPG6TF'FODIFM:PVOHJOFRVBMJUZ Ω⋆(θ) + Ω(y) ≥ {⟨θ, y⟩ − Ω(y)} + Ω(y) = ⟨θ, y⟩
(FPNFUSJDBM*OUFSQSFUBUJPO Ω(y) y μ 'PSSFHVMBSJ[FS
(FPNFUSJDBM*OUFSQSFUBUJPO Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU
BU CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ)
(FPNFUSJDBM*OUFSQSFUBUJPO Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU
BU CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ) ⟨θ, y⟩ − Ω⋆(θ) −Ω⋆(θ)
(FPNFUSJDBM*OUFSQSFUBUJPO Ω(y) y μ 'PSSFHVMBSJ[FS ̂ yΩ (θ) ESBXUBOHFOU
BU CZEFGPG'FODIFMDPOKVHBUF ̂ yΩ (θ) ⟨θ, μ⟩ − Ω⋆(θ) ⟨θ, y⟩ − Ω⋆(θ) −Ω⋆(θ) LΩ (y; θ) -PTTJTEJTUBODFCFUXFFO BOE BU #SFHNBOEJWFSHFODF y
&YBNQMF4IBOOPO&OUSPQZ HS (y) = − d ∑ j=1 yj
log yj ̂ yHS (θ) = argmax y∈Δd ⟨θ, y⟩ − HS (y) = exp θ ∑d j=1 exp θj TPGUNBY θ ̂ y(θ) CJOBSZTPGUNBYTJHNPJE LHS (θ; y) = H⋆ S (θ) + HS (y) − ⟨θ, y⟩ = log d ∑ j=1 exp θj − θk BTTVNJOHy = ek DSPTTFOUSPQZ JTMPHJTUJDMPTTJOCJOBSZDBTF LHS ( ̂ yHS (θ); y)
&YBNQMF5TBMMJT&OUSPQZ H2 (y) = 1 2 d ∑ j=1
yj (1 − yj ) Hα (y) = 1 α(α − 1) d ∑ j=1 (yj − yα j ) HS (y) = − d ∑ j=1 yj log yj 5TBMMJTFOUSPQZ α α → 2 α → 1 BLB(JOJJOEFY 4IBOOPOFOUSPQZ TQBSTFNBY θ ̂ y(θ) m ℓ(m) TQBSTFNBYMPTT H⋆ 2 (θ) + H2 (y) − ⟨θ, y⟩ NPEJpFE)VCFSMPTT TQFDJBMJ[FEJOCJOBSZDMBTTJpDBUJPO
0UIFS/JDF1SPQFSUZ 0WFSWJFX ˙ 4FQBSBUJPONBSHJO j j BpOJUFTDPSFBUUBJOT[FSPMPTT JG JTlTQBSTFz
˙ $BMJCSBUFETVSSPHBUF j NJOJNJ[JOH':MPTTMFBETUPNJOJNJ[JOHDMBTTJpDBUJPOFSSPS NPSFEJTDVTTJPOJTOFFEFEGPSTUSVDUVSFEQSFEJDUJPO ˙ &⒏DJFOUPQUJNJ[BUJPO j BMXBZTDPOWFYCZOBUVSFPQUJNJ[BCMFXJUI'SBOL8PMGFBMHPSJUIN JUFSBUJWFMZNJOJNJ[JOHMJOFBSBQQSPY Ω m ℓ(m) MPHJTUJD 4IBOOPO WBOJTIFTBU 㱣OPTFQNHO TQBSTFNBY 5TBMMJT TFQNHO m ℓ(m) 㱺OPQFOBMJ[BUJPOPOMBSHFFOPVHIQSFEJDUJPONBSHJOT
4VNNBSZ *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y UBSHFUMBCFM y
4VNNBSZ *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y UBSHFUMBCFM y ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) 3FHVMBSJ[FEQSFEJDUJPO NBLFTQBSTF USBDUBCMF ʜ
4VNNBSZ *OQVUTQBDF 4DPSFTQBDF 0VUQVUTQBDF x θ fW ̂ yΩ
QSFEJDUJPO ̂ y UBSHFUMBCFM y ̂ yΩ (θ) = argmax y∈dom(Ω) ⟨θ, y⟩ − Ω(y) 3FHVMBSJ[FEQSFEJDUJPO NBLFTQBSTF USBDUBCMF ʜ LΩ (θ; y) := Ω⋆(θ) + Ω(y) − ⟨θ, y⟩ 'FODIFM:PVOHMPTT TZTUFNBUJDXBZDPOTUSVDUJOHMPTTGSPNΩ