論文紹介 "Informer: Beyond Efficient Transformer for Long SequenceTime-Series Forecasting"

AAAI2021"Informer: Beyond Efficient Transformer for Long SequenceTime-Series Forecasting"の論文紹介用のスライドです。

1. ࿦จ঺հ Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting Taichi Murayama
2. 2 Title: Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting Author: Haoyi Zhou1, Shanghang Zhang2, Jieqi Peng1, Shuai Zhang1, Jianxin Li1, Hui Xiong3, Wancai Zhang4 (1 Beihang University (๺ژߤۭߤఱେֶ), 2 UC Berkeley, 3 Rutgers University, 4 SEDD Company) Conference: AAAI 2021 (Thirty-Fifth AAAI Conference on Artificial Intelligence) AAAI-21 Outstanding Paper! Bibliographic Information
3. 3 • ਓ޻஌ೳܥͷࠃࡍձٞͷτοϓΧϯϑΝϨϯε AAAIͷOutstanding Paper (͢͝ ͍ձٞͷதͰ΋͍͢͝࿦จ) • ௕ظతͳܥྻ༧ଌΛऔΓѻͬͨ࿦จ • Transformerͱ͸Ͳ͏͍͏΋ͷ͔Λཧղ Purpose: Why I choose this paper. ҰݴͰݴ͏ͱ… ʮ௕ظతͳܥྻ༧ଌΛ໨తͱͨ͠ɼܭࢉίετ΍ϝϞϦίετΛ ࡟ݮͨ͠TransformerͷมछΛఏҊ͠༧ଌੑೳΛ޲্ʯ • ܥྻ༧ଌλεΫʹ͓͍ͯTransformer(ଞͷਂ ૚ֶशϞσϧ΋ؚΉ)͕ۤखͳ఺͸Ͳ͜ͳͷ ͔ʁͲ͏͍͏ղܾࡦ͕͋Δ͔ʹ͍ͭͯཧղ
4. 4 Motivation • աڈͷܥྻ͔Β௕ظతͳকདྷͷܥྻ ྫ͑͹ϙΠϯτઌ΍ि ؒઌͷܥྻ Λ༧ଌ • ӳޠͰ͸಄จࣈΛऔͬͯ-45'ͱݺশ • طଘͷϞσϧͰ͸ࠔ೉ ྫ
   -45.ʹΑΔܥྻ༧ଌ औΓ૊Έ͍ͨ՝୊
   ௕ظతͳܥྻ༧ଌ -POH
   4FRVFODF
   5JNFTFSJFT
   'PSFDBTUJOH MSE score: ⻑期先の予測で 予測精度が低下 Inference speed :推論速度の低下
5. 5 Motivation • 5BTL
   աڈͷܥྻ͔Β௕ظతͳকདྷͷܥྻΛ༧ଌ͢Δͱ͍͏ܥྻ ༧ଌ໰୊ • *OQVU
   𝑋! = 𝑥" !, 𝑥# !, … , 𝑥$ ! 𝑥% ! ∈ ℝ&!} • 0VUQVU
   𝑌! = 𝑦" !, 𝑦# !, … , 𝑦$ ! 𝑦% ! ∈ ℝ&"} औΓ૊Έ͍ͨ՝୊
   ௕ظతͳܥྻ༧ଌ -POH
   4FRVFODF
   5JNFTFSJFT
   'PSFDBTUJOH
6. 6 -45'ʹऔΓ૊ΉͨΊͷͭͷ՝୊ • -POH
   4FRVFODF
   *OQVU
   -FBSOJOH
   1SPCMFN ௕ظతͳܥྻʹ͓͚Δ಺෦ؔ܎Λ ͲͷΑ͏ʹϞσϧͰଊ͑Δ͔ʁ ϝϞϦྔ໰୊ΛͲ͏ղܾ͢Δ͔ʁ • -POH
   4FRVFODF
   'PSFDBTUJOH
   1SPCMFN ೖྗͱग़ྗؒͷؔ܎ΛͲͷΑ͏ʹͯ͠ ଊ͑Δ͔ʁ Motivation
7. 7 -45'ʹର͢Δ5SBOTGPSNFS׆༻ͷ՝୊ • 5SBOTGPSNFSͷ௕ॴ
   • ࣗવݴޠॲཧ΍ը૾ॲཧͳͲʹݶΒͣܥྻ༧ଌʹ͓͍ͯ΋ߴ͍ਫ਼౓Λୡ੒ [Wu, 2020], [Yu, 2020] • ௕ظܥྻͷೖྗ಺Ͱͷؔ܎΍ΞϥΠϝϯτΛͱΔ͜ͱ͕Մೳ • 5SBOTGPSNFSͷ୹ॴ
   • ௕ظܥྻͷೖྗ΍ग़ྗͷܭࢉ΍ϝϞϦͷίετ͕ߴ্͘ख͍͔͘ͳ͍ • ௕ظܥྻͷਪ࿦ʹ͕͔͔࣌ؒΔ Motivation
8. 8 -45'ʹର͢Δ5SBOTGPSNFS׆༻ͷ՝୊ • 5SBOTGPSNFSͷ௕ॴ
   • ࣗવݴޠॲཧ΍ը૾ॲཧͳͲʹݶΒͣܥྻ༧ଌʹ͓͍ͯ΋ߴ͍ਫ਼౓Λୡ੒ [Wu, 2020], [Yu, 2020] • ௕ظܥྻͷೖྗ಺Ͱͷؔ܎΍ΞϥΠϝϯτΛͱΔ͜ͱ͕Մೳ • 5SBOTGPSNFSͷ୹ॴ
   • ௕ظܥྻͷೖྗ΍ग़ྗͷܭࢉ΍ϝϞϦͷίετ͕ߴ্͘ख͍͔͘ͳ͍ • ௕ظܥྻͷਪ࿦ʹ͕͔͔࣌ؒΔ Motivation ͦ΋ͦ΋5SBOTGPSNFSͱ͸Կ͔ʁ
9. 9 5SBOTGPSNFSͱ͸ʁ • (PPHMF͕ʮ"UUFOUJPO
   JT
   BMM
   ZPV
   OFFEʯͰఏҊ [Vaswani, 2017] • ໊લͷͱ͓Γɼ"UUFOUJPO͕ओཁͳߏ੒ཁૉ • ࣗવݴޠॲཧ /-1 ͚ͩͰͳ͘ɼ࠷ۙͰ͸$7 ͳͲͷଞ෼໺Ͱ΋༷ʑͳݚڀͰ׆༻ FY
   #&35
   (15
   7J5 • 5SBOTGPSNFSҎલ͸$//ͱ3//͕த৺ What’s Transformer?
10. 10 5SBOTGPSNFSҎલ 3FDVSSFOU
    /FVSBM
    /FUXPSL
    3// What’s Transformer? Value Times
11. 11 5SBOTGPSNFSҎલ 3FDVSSFOU
    /FVSBM
    /FUXPSL
    3// What’s Transformer? Value Times
12. 12 5SBOTGPSNFSҎલ $POWPMVUJPOBM
    /FVSBM
    /FUXPSL
    $// What’s Transformer? Value Times
13. 13 5SBOTGPSNFS What’s Transformer? Value Times Transformer
14. 14 5SBOTGPSNFSͷத਎ What’s Transformer? • جຊ͸&ODPEFS%FDPEFSϞσϧ Encoder Decoder
15. 15 5SBOTGPSNFSͷத਎ What’s Transformer? • جຊ͸&ODPEFS%FDPEFSϞσϧ • 1PTJUJPOBM
    &ODPEJOHͱ͍͏֤ೖྗͷ Ґஔ৘ใΛϕΫτϧͰදݱ
16. 16 5SBOTGPSNFSͷத਎ What’s Transformer? • جຊ͸&ODPEFS%FDPEFSϞσϧ • 1PTJUJPOBM
    &ODPEJOHͱ͍͏֤ೖྗͷ Ґஔ৘ใΛϕΫτϧͰදݱ • &ODPEFS
    %FDPEFSͱ΋ʹ .VMUJIFBE
    BUUFOUJPOͱ'FFE
    'PSXBSEͰ ߏ੒͞Εͨ5SBOTGPSNFS
    #MPDLͷੵΈॏͶ Transformer Block
17. 17 1PTJUJPOBM
    &ODPEJOH What’s Transformer? • ೖྗͷ֤UPLFOͷҐஔΛϕΫτϧͰදݱ • جຊ͸
    TJOؔ਺ͱDPTؔ਺Ͱ໌ࣔతʹఏڙ
18. 18 1PTJUJPOBM
    &ODPEJOH What’s Transformer? from: https://kazemnejad.com/blog/transformer_architecture_positional_encodin
19. 19 5SBOTGPSNFS
    #MPDL What’s Transformer? • &ODPEFSɼ%FDPEFSͱͱ΋ʹ5SBOTGPSNFS
    #MPDLͷੵΈॏͶͰߏ੒ • 5SBOTGPSNFS
    #MPDLͷߏ੒ 
    .VMUJ)FBE
    4FMGBUUFOUJPO 
    3FTJEVBM
    $POOFDUJPO
    ࢒ࠩ઀ଓ 
    -BZFS
    /PSNBMJ[BUJPO
    
    1PTJUJPOXJTF
    'FFE
    'PSXBSE 
    %SPQPVU Transformer Block
20. 20 4FMGBUUFOUJPO What’s Transformer? ⼊⼒系列の 潜在表現 系列⻑ × 次元数 Key : K Query: Q 𝑊' 𝑊( Value: V Attention Map : M 系列⻑ × 系列⻑ 𝑊) 𝑊*+! Output 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑄𝐾, 𝑑 )
21. 21 4FMGBUUFOUJPO What’s Transformer? Value: V Attention Map : M 系列⻑ × 系列⻑ 𝑊*+! Output 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑄𝐾, 𝑑 )
22. 22 .VMUJ)FBE
    4FMGBUUFOUJPO What’s Transformer? • ઌఔͷ4FMGBUUFOUJPOͷܭࢉΛෳ਺ฒߦ ࣮ͯ͠ߦ • ෳ਺ͷϕΫτϧʹ੾Γ෼͚ͯܭࢉ͢Δ͜ͱ Ͱೖྗؒͷଟ༷ͳྨࣅੑΛൃݟ͠ɼΑΓ ଟ༷ͳදݱྗΛ֫ಘ
23. 23 -45'ʹର͢Δ5SBOTGPSNFS׆༻ͷ՝୊ ࠶ܝ • 5SBOTGPSNFSͷ௕ॴ
    • ࣗવݴޠॲཧ΍ը૾ॲཧͳͲʹݶΒͣܥྻ༧ଌʹ͓͍ͯ΋ߴ͍ਫ਼౓Λୡ੒ [Wu, 2020], [Yu, 2020] • ௕ظܥྻͷೖྗ಺Ͱͷؔ܎΍ΞϥΠϝϯτΛͱΔ͜ͱ͕Մೳ • 5SBOTGPSNFSͷ୹ॴ
    • ௕ظܥྻͷೖྗ΍ग़ྗͷܭࢉ΍ϝϞϦͷίετ͕ߴ্͘ख͍͔͘ͳ͍ • ௕ظܥྻͷਪ࿦ʹ͕͔͔࣌ؒΔ Motivation ఏҊख๏*OGPSNFSͰղܾ
24. 24 5SBOTGPSNFSͷ$PNQVUBUJPOBM
    $PNQMFYJUZ Method Complexity per Layer Convolutional 𝑂 𝐾 / 𝐷! / 𝐿 Recurrent 𝑂 𝐿 / 𝐷! Self-attention (Transformer) 𝑂 𝐿! / 𝐷 Method: • ௕ظܥྻͰͳ͚Ε͹ %
    
    -ͱͳΓܭࢉ࣌ؒ͸ͦ͜·Ͱ໰୊ͳ͍͕ɼ ௕ظܥྻΛೖྗͱ͢Δͱɼ-
    
    %ͱͳΓܭࢉ࣌ؒ΍ϝϞϦ͕໰୊ʹ • ߋʹɼ-BZFSΛੵΈॏͶΔ͜ͱͰܭࢉ͕࣌ؒ/ഒʹ K: the length of filter D: dimensionality of space L: input length N: Number of layers Computational Complexityは 𝑶 𝑵×(𝑳𝟐 & 𝑫)
25. 25 *OGPSNFSʹΑΔ$PNQMFYJUZ࡟ݮ Method: ̎ͭͷ$PNQMFYJUZ࡟ݮख๏ΛఏҊ • 1SPC4QBSTF
    ॏཁ౓ͷߴ͍"UUFOUJPO
    .BQͷΈΛར༻ 𝑂 𝐿) # 𝐷 ˠ 𝑂 𝐿 log 𝐿 # 𝐷 ʹ࡟ݮ • 4FMGBUUFOUJPO
    %JTUJMMJOH
    ηϧϑΞςϯγϣϯ૚Λग़Δ౓ ܥྻͷ௕͕͞൒෼ʹͳΔΑ͏ৠཹ 𝑂 N # ⋯ ˠ 𝑂 2 − 𝜖 # ⋯ ʹ࡟ݮ
26. 26 4FMGBUUFOUJPO ࠶ܝ Method: ProbSparse ⼊⼒系列の 潜在表現 Key : K Query: Q 𝑊' 𝑊( Value: V Attention Map : M 系列⻑ × 系列⻑ 𝑊) 𝑊*+! Output 系列⻑(L) × 次元数(D) 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑄𝐾, 𝐷 )
27. 27 1SPC4QBSTF Method: ProbSparse ⼊⼒系列の 潜在表現 系列⻑(L) × 次元数(D) Key : K Query: 2 𝑄 𝑊' 𝑊( Value: V Attention Map : M 系列⻑ × 系列⻑ 𝑊) 𝑊*+! Output 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 8 𝑄𝐾, 𝐷 ) 上位u件のQuery のみを利⽤
28. 28 1SPC4QBSTF Method: ProbSparse ⼊⼒系列の 潜在表現 系列⻑(L) × 次元数(D) Key : K Query: 2 𝑄 𝑊' 𝑊( Value: V Attention Map: M 系列⻑ × 系列⻑ 𝑊) 𝑊*+! Output 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 8 𝑄𝐾⊺ 𝐷 ) 上位u件のQuery のみを利⽤ 1: その上位はどのように選択するのか？ 2: 計算効率はどれぐらい良くなるの？
29. 29 1SPC4QBSTF
    ॏཁ౓ͷߴ͍2VFSZͷબ୒ Method: ProbSparse 2VFSZ
    2ͷߦϕΫτϧ𝑞* ʹର͢Δ"UUFOUJPO
    .BQͷॏΈ ͜ͷॏΈ͕Ұ༷෼෍͔Βҳ୤͍ͯ͠Δ΄Ͳɼ ॏཁ౓͕ߴ͍
    4QBSTJUZ
    NFBTVSFNFOUͰࢉग़ Query: 2 𝑄 Attention Map : M 系列⻑ × 系列⻑ 9 5 𝑒𝑥𝑝 𝑞% , 𝑘5 𝐷 ∑6 𝑒𝑥𝑝 𝑞%, 𝑘6 𝐷 正規化項
30. 30 1SPC4QBSTF
    ॏཁ౓ͷߴ͍2VFSZͷબ୒ Method: ProbSparse "UUFOUJPO
    .BQࣗମ͕-POHUBJM
    %JTUSJCVUJPOͷੑ࣭Λ͍࣋ͬͯΔ
31. 31 1SPC4QBSTF
    ॏཁ౓ͷߴ͍2VFSZͷબ୒ Method: ProbSparse Ұ༷෼෍͔Β͔͚཭Ε͍ͯΔ͔Ͳ͏͔Ͱॏཁ౓͕ਪఆՄೳ
32. 32 1SPC4QBSTF
    ॏཁ౓ͷߴ͍2VFSZͷબ୒ Method: ProbSparse 4QBSTJUZ
    NFBTVSFNFOU Ұ༷෼෍ͱ"UUFOUJPO
    .BQͷ,VMMCBDL-FJCMFSڑ཭ʹج͍ͮͨई౓ ͜ͷई౓ͷ஋ʹج͍ͮͯɼ্ҐV݅ͷRVFSZͷΈΛར༻ 𝑀 𝑞! , 𝐾 = 𝑙𝑛 ( "#$ % 𝑒 &!'" ⊺ ( − 1 𝐿) ( "#$ % 𝑞! 𝑘" ⊺ 𝐷
33. 33 1SPC4QBSTF
    ॏཁ౓ͷߴ͍2VFSZͷબ୒ Method: ProbSparse 𝐾𝐿 𝑞 𝑝 = 1 𝐿 9 57" $ 𝑙𝑜𝑔 1 𝐿8" − 𝑙𝑜𝑔𝑍% + 𝑞%𝑘5 𝐷 = 𝑙𝑜𝑔 1 𝐿8" − 𝑙𝑜𝑔𝑍% + 1 𝐿 9 57" $ 𝑞%𝑘5 𝐷 = 𝑙𝑜𝑔 1 𝐿8" − 𝑙𝑜𝑔 9 57" $ exp( 𝑞%𝑘5 𝐷 ) + 1 𝐿 9 57" $ 𝑞%𝑘5 𝐷 ⼀様分布とAttention MapのKL距離 𝑞 = 1 𝐿 𝑝 = 𝑒𝑥𝑝 𝑞"𝑘# 𝐷 ∑$ 𝑒𝑥𝑝 𝑞", 𝑘$ 𝐷 = 𝑒𝑥𝑝 𝑞"𝑘# & 𝐷% & ' 𝑍" ⼀様分布 Query 𝒒𝒊 における Attention Map 𝑀 𝑞", 𝐾 = 𝑙𝑛 ; #(& )! 𝑒 *"+# ⊺ , − 1 𝐿- ; #(& )! 𝑞"𝑘# ⊺ 𝐷 Sparsity Measurement 定数項 -1 × Sparsity Measurement ref: https://cookie-box.hatenablog.com/entry/2021/02/11/195
34. 34 1SPC4QBSTF
    ܭࢉޮ཰ͷ໰୊ Method: ProbSparse 4QBSTJUZ
    .FBTVSFNFOUͷܭࢉࣗମ͕𝑶 𝑳𝟐 ͜ͷܭࢉࣜΛҎԼͷΑ͏ʹLFZ
    WFDUPS͔ΒαϯϓϦϯάʹΑΔۙࣅ Λߦ͏͜ͱͰ 𝑂 L log 𝐿 Λୡ੒ (Max − mean Measurementͷܗ) 𝑀 𝑞% , 𝐾 = 𝑙𝑛 9 57" $# 𝑒 :$;% ⊺ & − 1 𝐿' 9 57" $# 𝑞% 𝑘5 ⊺ 𝐷 L 𝑀 𝑞% , 𝐾 = max 5 𝑞% 𝑘5 ⊺ 𝐷 − 1 𝐿' 9 57" $# 𝑞% 𝑘5 ⊺ 𝐷
35. 35 ,-ڑ཭≥ 0Ͱ͋Δ͜ͱ͔Βɼ𝐾𝐿 𝑞 𝑝 = 𝑙𝑜𝑔 > ?!" − 𝑀 𝑞* , 𝐾 ΑΓ ·ͨҎԼͷࣜʹΑΓ 1SPC4QBSTF
    ܭࢉޮ཰ͷ໰୊ Method: ProbSparse 𝑀 𝑞%, 𝐾 = 𝑙𝑛 9 57" $# 𝑒 :$;% ⊺ & − 1 𝐿' 9 57" $# 𝑞% 𝑘5 ⊺ 𝐷 ≤ 𝑙𝑛 𝐿 P max 5 𝑒 :$;% ⊺ & − 1 𝐿' 9 57" $# 𝑞% 𝑘5 ⊺ 𝐷 = 𝑙𝑛𝐿 + max 5 :$;% ⊺ & − " $# ∑ 57" $# :$;% ⊺ & 𝑙𝑜𝑔𝐿 ≤ 𝑀 𝑞! , 𝐾 = G 𝑀 𝑞* , 𝐾 : Sparsity Measurementの近似解
36. 36 1SPC4QBSTF
    ΞϧΰϦζϜ Method: ProbSparse Sampling Sparsity Measurement Select Top-u from Q Calculate Attention Map
37. 37 *OGPSNFSʹΑΔ$PNQMFYJUZ࡟ݮ ࠶ܝ Method: ̎ͭͷ$PNQMFYJUZ࡟ݮख๏ΛఏҊ • 1SPC4QBSTF
    ॏཁ౓ͷߴ͍"UUFOUJPO
    .BQͷΈΛར༻ 𝑂 𝐿) # 𝐷 ˠ 𝑂 𝐿 log 𝐿 # 𝐷 ʹ࡟ݮ • 4FMGBUUFOUJPO
    %JTUJMMJOH
    ηϧϑΞςϯγϣϯ૚Λग़Δ౓ ܥྻͷ௕͕͞൒෼ʹͳΔΑ͏ৠཹ 𝑂 N # ⋯ ˠ 𝑂 2 − 𝜖 # ⋯ ʹ࡟ݮ
38. 38 Self-attention Distilling Method: Self-attention Distilling ೖྗ͕4FMGBUUFOUJPO૚Λग़Δͨͼʹɼܥྻͷ௕͕͞൒෼ʹͳΔΑ͏ʹৠཹ ৠཹࣗମ͸෺ମݕग़[Yu, 2017]ͳͲʹ΋༻͍ΒΕ͍ͯΔܰྔԽख๏ͷ1छ Attention Blockから でるたびに蒸留
39. 39 Self-attention Distilling Method: Self-attention Distilling K൪໨ͷK ൪໨ͷϨΠϠʔʹೖྗ͢Δͱ͖ʹɼ$POWE
    LFSOFM
    XJEUI ͱ.BY1PPMJOHΛ௨͢͜ͱͰܥྻ௕Λѹॖ ϨΠϠʔ਺෼͔ΒഒҎԼͷ$PNQMFYJUZʹѹॖ 𝑂 N # ⋯ ˠ 𝑂 2 − 𝜖 # ⋯ 𝑋@A> B = 𝑀𝑎𝑥𝑃𝑜𝑜𝑙 𝐸𝐿𝑈 𝐶𝑜𝑛𝑣1𝑑 𝑋@ B
40. 40 Method: Decoder Outputs through one forward "VUPSFHSFTTJWFʹΑΔਪ࿦ [Chen, 2019] /POBVUPSFHSFTTJWFʹΑΔਪ࿦ ճؼʹΑΒͳ͍ਪ࿦
41. 41 Method: Decoder Outputs through one forward "VUPSFHSFTTJWFʹΑΔਪ࿦ [Chen, 2019] /POBVUPSFHSFTTJWFʹΑΔਪ࿦ ճؼʹΑΒͳ͍ਪ࿦ Informerはこれを採⽤ 推論が早くなるというメリット
42. 42 Experiment: Dataset • &5'
    தࠃͷͭͷ஍Ҭͷిྗڙڅͷσʔλ BVUIPST
    DPMMFDU
    • ͭͷܥྻ • ࣌ؒ୯Ґͱ෼୯Ґͷه࿥ • 5SBJOWBMUFTU
    
    NPOUIT • &$-
    ਓͷిྗফඅͷσʔλ [Li, 2019] • ࣌ؒ୯Ґͷه࿥ • 5SBJOWBMUFTU
    
    NPOUIT • 8FBUIFS
    ΞϝϦΧ߹ऺࠃͷ஍఺ͷؾީσʔλ • ࣌ؒ୯Ґͷه࿥ • 5SBJOWBMUFTU
    
    NPOUIT
43. 43 Experiment: Baseline and Evaluation Metric • #BTFMJOF • ARIMA • Prophet [Taylor, 2018] • LSTMa [Bahdanau, 2014] • LSTnet [Lai, 2018] • DeepAR [Salinas, 2020] • LogTrans [Li, 2019] • Reformer [Kitaev, 2019] • &WBMVBUJPO
    .FUSJDT • Mean Squared Error (MSE): " < ∑%7" < 𝑦 − Q 𝑦 # • Mean Absolute Error (MAE): " < ∑%7" < 𝑦 − Q 𝑦
44. 44 Result: Univariate Time-series Forecasting • Informerが多くのデータセット+ 予測先において⾼い精度を達成 (特に⻑期予測) • Query Sparsityを採⽤しなかったInformerと⽐べても，Informerが⾼い精度を達成 Attentionが着⽬する場所を制限することの効果を⽰す
45. 45 Result: Multivariate Time-series Forecasting • 多変量でもInformerが多くのデータセット+ 予測先において⾼い精度を
46. 46 Result: Multivariate Time-series Forecasting
47. 47 1SPC4QBSTFʹΑͬͯ௕ظ༧ଌ͕ϝϞϦ໰୊ʹࠔΒͣɼߴ͍ਫ਼౓Ͱ ༧ଌΛୡ੒ Result: Ablation Study "CMBUJPO
    PG
    1SPC4QBSTF
48. 48 4FMGBUUFOUJPO
    %JTUJMMJOH΋ಉ༷ʹϝϞϦ໰୊ʹࠔΒͣɼ௕ظ༧ଌ͕ Մೳʹ Result: Ablation Study "CMBUJPO
    PG 4FMGBUUFOUJPO
    %JTUJMMJOH
49. 49 4FMGBUUFOUJPO
    %JTUJMMJOH΋ಉ༷ʹϝϞϦ໰୊ʹࠔΒͣɼ௕ظ༧ଌ͕ Մೳʹ Result: Ablation Study "CMBUJPO
    PG 4FMGBUUFOUJPO
    %JTUJMMJOH
50. 50 /POBVUPSFHSFTTJWFͳग़ྗʹΑͬͯ༧ଌޡࠩͷੵΈॏͶΛແࢹ Result: Ablation Study "CMBUJPO
    PG
    HFOFSBUJWF
    TUZMF
    EFDPEFS 推論速度も⻑期予測に おいて顕著な差
51. 51 • ௕ظతͳܥྻ༧ଌΛ໨తͱͨ͠ɼܭࢉίετ΍ϝϞϦίετΛ ࡟ݮͨ͠Transformerͷมछ: InformerΛఏҊ͠༧ଌੑೳΛ޲্ • ॏཁ౓ͷߴ͍෦෼ͷΈΛAttention͢ΔProbSparseͱৠཹख๏Λ׆ ༻ͨ͠Self-attention DistillingʹΑͬͯϝϞϦίετΛ࡟ݮ • Non-autoregressiveͳग़ྗΛߦ͏͜ͱͰ௕ظ༧ଌʹ͓͍ͯ΋ਫ਼౓ Λҡ࣋ͨ͠ૣ͍ਪ࿦Λ࣮ݱ • ిྗσʔλ΍ؾީσʔλͳͲͷσʔλͰInformerͷ༗ޮੑΛࣔ͢ Result: Summary
52. 52 [Wu, 2020] Wu, Neo, et al. "Deep transformer models for time series forecasting: The influenza prevalence case." arXiv preprint arXiv:2001.08317 (2020). [Yu, 2020] Yu, Cunjun, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. "Spatio-temporal graph transformer networks for pedestrian trajectory prediction." In European Conference on Computer Vision, pp. 507-523. Springer, Cham, 2020. [Vaswani, 2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008). [Yu, 2017] Yu, Fisher, Vladlen Koltun, and Thomas Funkhouser. "Dilated residual networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. [Chen, 2019] Chen, Nanxin, et al. "Listen and fill in the missing letters: Non-autoregressive transformer for speech recognition." arXiv preprint arXiv. [Li, 2019] Li, Shiyang, et al. "Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting." Advances in Neural Information Processing Systems 32 (2019): 5243-5253. [Taylor, 2018] Taylor, Sean J., and Benjamin Letham. "Forecasting at scale." The American Statistician 72.1 (2018): 37-45. [Bahdanau, 2014] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014). [Lai, 2018] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018. [Salinas, 2020] Salinas, David, et al. "DeepAR: Probabilistic forecasting with autoregressive recurrent networks." International Journal of Forecasting 36.3 (2020): 1181-1191. [Li, 2019] Li, S.; Jin, X.; Xuan, Y.; Zhou, X.; Chen, W.; Wang, Y.-X.; and Yan, X. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Fore- casting. arXiv:1907.00235 . [Kitaev, 2019] Kitaev, N.; Kaiser, L.; and Levskaya, A. 2019. Reformer: The Efficient Transformer. In ICLR. Reference