Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
文献紹介 1月24日
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
gumigumi7
January 24, 2019
260
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
文献紹介 1月24日
Pay Less Attention with Lightweight and Dynamic Convolutions
gumigumi7
January 24, 2019
Transcript
17 slides
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli,
International Conference on Learning Representations, 2019
%1 2 n Transformer self-attention -# )6( ;7+ 0'&:* n
'$.4 SotA8" (ICLR2019 3 !/) n 251, 0!/39
3 n RNNCNNself-Attention.( Sequence Modeling"& % n +
4*(self-attention))'05 ( $, l Ex. ) Transformer l self-attention -!/ #31 2
$ 4 n ) 0%'2 8(# 417! n 417!(#$)
88"3*9! n 0%'2 +&/ (Tang et al., 2018) n .6-,5
5 n ) Self-attention l "!
"# n ) Dynamic convolution () l $ "
6 n Self-attention n Gated linear units
(GLU) Lightweight conv() n Dynamic conv
7 n Self-attention n
8 n Depthwise convolutions n "! ! n
# we have to go to Tokyo tonight we have to go to Tokyo tonight Normal convolutions Depthwise convolutions
9 n Lightweight convolutions n
n Softmax we have to go to Tokyo tonight Lightweight convolutions
10 n & ' " Dynamic convolutions
n $ # # & ( ' !" & ' " n %# $!self-attention
11 n Encoder-Decoder n Transformer self-attentionLightweight Conv,
Dynamic Conv n
12
() 13 • En-De, En-Fr self-attention (Vaswani
et al., 2017) SotA • Zn-En
3+ (48) 14 • - 9 • :%CNN 0'(5
(CNN, k=3) • Kernel$# /1&72 $(5! • Softmax;*,6 " 3+.)
( ) 15 • • Self-attention
-( (0,) 16 • Self-attention" 4&/13 • Bottom-Up
0, ) sequence-to-sequence • $!.*(Celikyilmaz et al., 2018) +# $!.LightConv, DynamicConv 2* / 4&/'%
17 n Self-attention ) 5$-' ,+28 ! . n
# # 64;SotA9( n 7&31/ : "&0*%
More Decks by gumigumi7
See All by gumigumi7
文献紹介 11月7日
gumigumi7
0
140
文献紹介 10月3日
gumigumi7
0
340
文献紹介 9月3日
gumigumi7
0
270
文献紹介 8月10日
gumigumi7
0
140
文献紹介 7月16日
gumigumi7
0
280
文献紹介 6月12日
gumigumi7
0
350
文献紹介 5月16日
gumigumi7
0
200
文献紹介 4月18日
gumigumi7
0
150
文献紹介 12月15日
gumigumi7
0
130
Featured
See All Featured
Accessibility Awareness
sabderemane
1
310
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.6k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
17k
Odyssey Design
rkendrick25
PRO
3
960
How to build a perfect <img>
jonoalderson
1
6.2k
Lessons Learnt from Crawling 1000+ Websites
charlesmeaden
PRO
1
1.8k
Agile that works and the tools we love
rasmusluckow
331
22k
Navigating Weather and Climate Data
rabernat
0
590
The Illustrated Children's Guide to Kubernetes
chrisshort
51
53k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
rkaga
63
48k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
greggifford
PRO
0
400
Bash Introduction
62gerente
615
220k