decision trees. A feature vector is classified by descending each tree. This gives, lass distribution at the leaf. As an illustration, we highlight the root to leaf paths (yellow) ure vector. This paper shows how to simultaneously exploit both the hierarchical clustering ss distributions. (b) Semantic texton forests features. The split nodes in semantic texton els within a d d patch: either the raw value of a single pixel, or the sum, difference, or mentation by em- present. of this work are: ntly provide both xtons and a local xtons model, and entation; and (iii) ove segmentation ion 2 gives a brief ch form the basis uced in Section 3. in Section 4 and ing, we also use it as a classifier, which enables us to use semantic context for image segmentation; and (iii) in addi- tion to the leaf nodes used in [19], we include the branch nodes as hierarchical clusters. A related method, the pyra- mid match kernel [9], exploits a hierarchy in descriptor space, though it requires the computation of feature descrip- tors and is only applicable to kernel based classifiers. The pixel-based features we use are similar to those in [14], but our forests are trained to recognize object categories, not match particular feature points. Other work has also looked at alternatives to k-means. Recent work [29] quantizes feature space into a hyper-grid, but requires descriptor computation and can result in very uilding rass ee ow heep ky irplane ater ce ar icycle ower ign ird ook hair ad at og ody oat lobal verage .BDIJOFMFBSOJOHBMHPSJUIN )BOEDSBGUFEGFBUVSF Figure 2. (a) Decision forests. A forest consists of T decision trees. A feature vector is classified by descending each tree. This gives, for each tree, a path from root to leaf, and a class distribution at the leaf. As an illustration, we highlight the root to leaf paths (yellow) and class distributions (red) for one input feature vector. This paper shows how to simultaneously exploit both the hierarchical clustering implicit in the tree structure and the node class distributions. (b) Semantic texton forests features. The split nodes in semantic texton forests use simple functions of raw image pixels within a d d patch: either the raw value of a single pixel, or the sum, difference, or absolute difference of a pair of pixels (red). act as an image-level prior to improve segmentation by em- phasizing the categories most likely to be present. To summarize, the main contributions of this work are: (i) semantic texton forests which efficiently provide both a hierarchical clustering into semantic textons and a local classification; (ii) the bag of semantic textons model, and its applications in categorization and segmentation; and (iii) the use of the image-level prior to improve segmentation performance. The paper is structured as follows. Section 2 gives a brief recap of randomized decision forests which form the basis of our new semantic texton forests, introduced in Section 3. ing, we also use it as a classifier, which enables us to use semantic context for image segmentation; and (iii) in addi- tion to the leaf nodes used in [19], we include the branch nodes as hierarchical clusters. A related method, the pyra- mid match kernel [9], exploits a hierarchy in descriptor space, though it requires the computation of feature descrip- tors and is only applicable to kernel based classifiers. The pixel-based features we use are similar to those in [14], but our forests are trained to recognize object categories, not match particular feature points. Other work has also looked at alternatives to k-means. Recent work [29] quantizes feature space into a hyper-grid, 3BOEPNGPSFTU 1JYFMDPNQBSJTPO
H′  H W W H ըૉ͝ͱʹΫϥε֬Λग़ྗ W H 1FSTPO 1FSTPO 1FSTPO 1FSTPO C ʜ ʜ ɿΈࠐΈ ɿϓʔϦϯά ɿΞοϓαϯϓϦϯά C άϦου͝ͱʹ Ϋϥε֬ͱݕग़ྖҬΛग़ྗ Ϋϥε֬Λग़ྗ ೖྗ ग़ྗ C + B $// ग़ྗ݁ՌͷՄࢹԽ $// $// ମݕग़ ը૾ྨ ηϚϯςΟοΫ ηάϝϯςʔγϣϯ
SegNet predictions on urban and highway scene test samples from the wild. The class colour codes can be obtained from Brostow et al. [3]. To try our system yourself, please see our online web demo at http://mi.eng.cam.ac.uk/projects/segnet/. 1PPMJOHJOEJDFT ϓʔϦϯάҐஔΛهԱ͠ɼΞοϓαϯϓϦϯά࣌ʹར༻
Pixel Representations Pixel-Region Relation Object Contextual Representations Augmented Representations N N N Soft Object Regions Object Region Representations Loss ΦϒδΣΫτΫϥεͷදݱ Λར༻ͯ͠ϐΫηϧ͝ͱͷ ಛΛ֫ಘ ΦϒδΣΫτྖҬʹଘࡏ͢Δ ϐΫηϧͷදݱΛूू ΦϒδΣΫτྖҬͷදݱΛՃॏूͨ͠ ΦϒδΣΫτίϯςΩετදݱ "411 0$3 ιϑτΞςϯγϣϯ
⨂ SE module A C×N N×1 SE module B SE module C ⨂ C×H×W C×H×W C×1 C×1 C×1 C×1 C×1 Domain Assignment softmax FC GAP SE Adapter C×H×W Concat. GAP υϝΠϯ# 3FTJEVBM #MPDL υϝΠϯ" υϝΠϯ$ υϝΠϯʹର͢ΔΞςϯγϣϯ
B SE module C ⨂ C×H×W C×H×W C×1 C×1 C×1 C×1 C×1 Domain Assignment softmax FC GAP SE Adapter C×H×W Concat. GAP ⨂ SE module A C×N N×1 SE module B SE module C ⨂ C×H×W C×H×W C×1 C×1 C×1 C×1 C×1 Domain Assignment softmax FC GAP SE Adapter C×H×W C×H×W GAP Concat. υϝΠϯ# 3FTJEVBM #MPDL υϝΠϯ" υϝΠϯ$ w %PNBJO"UUFOUJPO %" NPEVMF 4&"EBQUFSɿ֤ϒϥϯνಛఆͷυϝΠϯใΛநग़ %PNBJO"TTJHONFOUɿυϝΠϯʹର͢ΔΞςϯγϣϯΛٻΊॏΈ͚ υϝΠϯʹର͢ΔΞςϯγϣϯ
module A C×N N×1 SE module B SE module C ⨂ C×H×W C×H×W C×1 C×1 C×1 C×1 C×1 Domain Assignment softmax FC GAP SE Adapter C×H×W C×H×W GAP Concat. ఏҊख๏ɿ.VMUJ%PNBJO4FNBOUJD4FHNFOUBUJPOVTJOH.VMUJ)FBE.PEFM υϝΠϯ# 3FTJEVBM #MPDL υϝΠϯ" υϝΠϯ$ ˠυϝΠϯ͝ͱʹ4&"EBQUFS͔Β࠷దͳಛΛ֫ಘ υϝΠϯʹର͢ΔΞςϯγϣϯ
7BMVF Ͱߏ 4DBMFE%PU1SPEVDU"UUFOUJPO Attention(Q, K, V) = softmax( QKT dk )V Scaled Dot-Product Attention Multi-Head Attention Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. 3.2.1 Scaled Dot-Product Attention We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys, divide each by p dk , and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as: Attention(Q, K, V ) = softmax( QKT p dk )V (1) The two most commonly used attention functions are additive attention [2], and dot-product (multi- ఆࣜԽ 2VFSZͷ࣍ݩ dk ྫɿ2 ,ͷฏۉ͕ɼࢄ͕ͱԾఆ͢Δͱɼ͜ΕΒͷߦྻੵͷฏۉ͕ɼࢄ ͱͳΔ dk 4PGUNBYؔͷޯΛܭࢉ࣌ʹɼҰ෦ͷੵ͕ඇৗʹେ͖͍ͱɼ ੵ͕࠷େͷཁૉҎ֎ͷޯ͕ඇৗʹখ͘͞ͳΔ 2 ,ͷಛྔΛ ͰεέʔϦϯά͢Δ͜ͱͰɼฏۉɼࢄͱͳΓΒ͔ͳޯΛͱΔ dk Scaled Dot-Product Attention Multi-Head Attention Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. 3.2.1 Scaled Dot-Product Attention
q4 k4 v4 q5 k5 v5 x1 e1 x2 e2 x3 e3 x4 e4 x5 e5 &NCFEEJOH &NCFEEJOH &NCFEEJOH &NCFEEJOH &NCFEEJOH 1& Figure 1: The Transformer - model architecture. der and Decoder Stacks Figure 1: The Transformer - model architecture. 1& Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Figure 1: The Transformer - model architecture. 1& Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Figure 1: The Transformer - model architecture. 1& Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Figure 1: The Transformer - model architecture. 1& Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Figure 1: The Transformer - model architecture. Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Encoder: The encoder is composed of a stack of N = 6 identical layers. Each qi = Wq ei ki = Wk ei vi = Wv ei &NCFEEJOHͨ͠ಛྔ ͔Β2VFSZɼ,FZɼ7BMVFಛྔΛͦΕͧΕͷઢܗมͰٻΊΔ ei 2VFSZ ,FZ 7BMVF
ن Խ ઢ෦Λ/ճ܁Γฦ͢ ಛϕΫτϧ lࢲz lz lݘ l͕z l͖z lͩz lɻz &ODPEFS .BTLFE .VMUJ)FBE "UUFOUJPO ਖ਼ ن Խ .VMUJ)FBE "UUFOUJPO ਖ਼ ن Խ ಛϕΫτϧ GFFEGPSXBSE ਖ਼ ن Խ ྨ ث l*z &04 .BTLFE .VMUJ)FBE "UUFOUJPO ਖ਼ ن Խ ਖ਼ ن Խ ಛϕΫτϧ GFFEGPSXBSE ਖ਼ ن Խ ྨ ث lMJLFz &04l*z .BTLFE .VMUJ)FBE "UUFOUJPO ਖ਼ ن Խ ਖ਼ ن Խ ಛϕΫτϧ GFFEGPSXBSE ਖ਼ ن Խ ྨ ث &/% &04l*zlMJLFzlEPHTzlz ઢ෦Λ/ճ܁Γฦ͢ ઢ෦Λ/ճ܁Γฦ͢ ઢ෦Λ/ճ܁Γฦ͢ %FDPEFS ʜ Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position- wise fully connected feed-forward network. We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512. Decoder: The decoder is also composed of a stack of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization. We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i. 3.2 Attention An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. 3 2 , 7 .VMUJ)FBE "UUFOUJPO 2 , 7 .VMUJ)FBE "UUFOUJPO 2 , 7
Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them,
Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder Preprint. Under review. Transformer Encoder MLP Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, Preprint. Under review. Transformer Encoder MLP Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, Preprint. Under review. Transformer Encoder MLP Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them,
arns em- em- the idal That ex- ield tire the the nte- ten- find west y is &NCFEEJOH 'MBUUFO x1 p x2 p x3 p x4 p x1 p E x2 p E x3 p E x4 p E E ∈ ℝ(P2⋅C)×D <DMT>UPLFO xN p ∈ ℝP2⋅C + + + + + 1PTJUJPO&NCFEEJOH Epos ∈ ℝ(N+1)×D 7J5 ඍՄೳͳύϥϝʔλͰ ໌ࣔతʹҐஔڭ͑ͣʹࣗಈͰֶश͞ΕΔ ࢝ ϕΫτϧͰදݱ Ͱ͋Γ ը૾ಛ͕7J5Ͱू͞Ε࠷ऴͰΫϥεྨ 5SBOTGPSNFSͷ&NCFEEJOHಛྔͱಉ͡ѻ͍ N ɿύον ɿ࣍ݩ D ɿνϟϯωϧ ɿύονը૾αΠζ C P ೖྗը૾ΛYʹখ͚ ʜ ʜ Epos ϕʔεϞσϧج४
arns em- em- the idal That ex- ield tire the the nte- ten- find west y is &NCFEEJOH 'MBUUFO x1 p x2 p x3 p x4 p x1 p E x2 p E x3 p E x4 p E E ∈ ℝ(P2⋅C)×D <DMT>UPLFO xN p ∈ ℝP2⋅C + + + + + 1PTJUJPO&NCFEEJOH Epos ∈ ℝ(N+1)×D ඍՄೳͳύϥϝʔλͰ ໌ࣔతʹҐஔڭ͑ͣʹࣗಈͰֶश͞ΕΔ ࢝ ϕΫτϧͰදݱ Ͱ͋Γ ը૾ಛ͕7J5Ͱू͞Ε࠷ऴͰΫϥεྨ 5SBOTGPSNFSͷ&NCFEEJOHಛྔͱಉ͡ѻ͍ N ɿύον ɿ࣍ݩ D ɿνϟϯωϧ ɿύονը૾αΠζ C P ೖྗը૾ΛYʹখ͚ ʜ ʜ Epos ϕʔεϞσϧج४ Preprint. Under review. Figure 7: Left: Filters of the initial linear embedding of RGB values of ViT-L/32. Center: Sim- ilarity of position embeddings of ViT-L/32. Tiles show the cosine similarity between the position $//ͷ࣍ݩͷϑΟϧλͷΑ͏ͳͷֶ͕श͞ΕΔ 7J5
the arns em- em- the idal That ex- ield tire the the nte- ten- find west y is &NCFEEJOH 'MBUUFO x1 p x2 p x3 p x4 p x1 p E x2 p E x3 p E x4 p E E ∈ ℝ(P2⋅C)×D <DMT>UPLFO xN p ∈ ℝP2⋅C + + + + + 1PTJUJPO&NCFEEJOH Epos ∈ ℝ(N+1)×D ඍՄೳͳύϥϝʔλͰ ໌ࣔతʹҐஔڭ͑ͣʹࣗಈͰֶश͞ΕΔ ࢝ ϕΫτϧͰදݱ Ͱ͋Γ ը૾ಛ͕7J5Ͱू͞Ε࠷ऴͰΫϥεྨ 5SBOTGPSNFSͷ&NCFEEJOHಛྔͱಉ͡ѻ͍ N ɿύον ɿ࣍ݩ D ɿνϟϯωϧ ɿύονը૾αΠζ C P ೖྗը૾ΛYʹখ͚ ʜ ʜ Epos ϕʔεϞσϧج४ Preprint. Under review. Figure 7: Left: Filters of the initial linear embedding of RGB values of ViT-L/32. Center: Sim- ͋Δ1&ͱଞͷ1&ͱͷྨࣅΛදݱͨ͠ͷ ۙͷଞͷҐஔͱͷྨࣅ͕ߴ͍ ͔ΒԕํͷଞͷҐஔͱͷྨࣅ͕͍ ۙͷ1&ಉ࢜ࣅͨʹͳΔΑ͏ʹֶश͞Ε͍ͯΔ
w 5SBOTGPSNFSہॴత͔ͭҬతͳಛΛऔಘՄೳ w .-1ہॴతͳಛΛิͰ͖ɼڧྗͳදݱΛ֫ಘՄೳ 7J5ϕʔεͷηάϝϯςʔγϣϯɿ4FH'PSNFS<9JF BS9JW> &9JF l4FH'PSNFS4JNQMFBOE& ff i DJFOU%FTJHOGPS4FNBOUJD4FHNFOUBUJPOXJUI5SBOTGPSNFST zBS9JW Overlap Patch Embeddings Transformer Block 1 MLP Layer ! " × # " ×"$ ! % × # % ×"& ! '& × # '& ×"" ! $( × # $( ×"' ! " × # " ×4" MLP ! " × # " ×$)*+ Transformer Block 2 Transformer Block 3 Transformer Block 4 Overlap Patch Merging Efficient Self-Attn Mix-FFN ×" UpSample MLP ! "!"# × # "!"# ×"$ ! "!"# × # "!"# ×" ! % × # % ×" Encoder Decoder Figure 2: The proposed SegFormer framework consists of two main modules: A hierarchical Transformer encoder to extract coarse and fine features; and a lightweight All-MLP decoder to directly fuse these multi-level features and predict the semantic segmentation mask. “FFN” indicates feed-forward network. i the MiT encoder go through an MLP layer to unify the channel dimension. Then, in a second step, features are up-sampled to 1/4th and concatenated together. Third, a MLP layer is adopted to fuse the concatenated features F. Finally, another MLP layer takes the fused feature to predict the segmentation mask M with a H 4 ⇥ W 4 ⇥ N cls resolution, where N cls is the number of categories. This lets us formulate the decoder as: ˆ F i = Linear(C i , C)(F i ), 8i ˆ F i = Upsample( W 4 ⇥ W 4 )( ˆ F i ), 8i F = Linear(4C, C)(Concat( ˆ F i )), 8i M = Linear(C, N cls )(F), (4) where M refers to the predicted mask, and Linear(C in , C out )(·) refers to a linear layer with C in and C out as input and output vector dimensions respectively. DeepLabv3+ SegFormer Stage-1 Stage-2 Stage-3 Head Stage-4 Figure 3: Effective Receptive Field (ERF) on Cityscapes (aver- age over 100 images). Top row: Deeplabv3+. Bottom row: Seg- Former. ERFs of the four stages and the decoder heads of both Effective Receptive Field Analysis. For semantic segmentation, maintain- ing large receptive field to include con- text information has been a central is- sue [5, 19, 20]. Here, we use effec- tive receptive field (ERF) [70] as a toolkit to visualize and interpret why our MLP decoder design is so effec- tive on Transformers. In Figure 3, we visualize ERFs of the four encoder stages and the decoder heads for both 3FE#PYɿ4UBHFͷ4FMG"UUFOUJPOͷSFDFQUJWF fi FMEͷେ͖͞ #MVF#PYɿ.-1-BZFSͷ.-1ͷSFDFQUJWF fi FMEͷେ͖͞ $JUZTDBQFTʹ͓͚ΔSFDFQUJWF fi FMEͷޮՌΛੳ ˠ$//ϕʔεہॴతͳಛͷΈଊ͑Δ͕ɼ5SBOTGPSNFSϕʔεہॴతɾҬతͳಛΛଊ͑Δ ˠ.-1ͷSFDFQUJWF fi FMEہॴతྖҬͷີ͕ߴ͍ খ͍͞ମͷਖ਼֬ͳηάϝϯςʔγϣϯ͕ظ
w 4FH'PSNFSͷޮՌ 7JTJPO5SBOTGPSNFSͷʹΑΔରϊΠζੑͷ্ ·ͱΊɿਂֶशʹΑΔηϚϯςΟοΫηάϝϯςʔγϣϯ ResNet101 + DA module ASPP 1×1conv. concat Cityscapes !" !# !$ マルチヘッド構造 3×3conv. 3×3conv. 1×1conv. Head 1 %" %# %$ Head 2 Head & 3×3conv. 3×3conv. 1×1conv. 3×3conv. 3×3conv. 1×1conv. Cityscapes 共有ネットワーク A2D2 Mapillary A2D2 Mapillary Overlap Patch Embeddings Transformer Block 1 MLP Layer ! " × # " ×"$ ! % × # % ×"& ! '& × # '& ×"" ! $( × # $( ×"' ! " × # " ×4" MLP ! " × # " ×$)*+ Transformer Block 2 Transformer Block 3 Transformer Block 4 Overlap Patch Merging Efficient Self-Attn Mix-FFN ×" UpSample MLP ! "!"# × # "!"# ×"$ ! "!"# × # "!"# ×" ! % × # % ×" Encoder Decoder Figure 2: The proposed SegFormer framework consists of two main modules: A hierarchical Transformer encoder to extract coarse and fine features; and a lightweight All-MLP decoder to directly fuse these multi-level features and predict the semantic segmentation mask. “FFN” indicates feed-forward network. in an end-to-end manner. After that, researchers focused on improving FCN from different aspects such as: enlarging the receptive field [17–19, 5, 2, 4, 20]; refining the contextual information [21– 29]; introducing boundary information [30–37]; designing various attention modules [38–46]; or using AutoML technologies [47–51]. These methods significantly improve semantic segmentation performance at the expense of introducing many empirical modules, making the resulting framework computationally demanding and complicated. More recent methods have proved the effectiveness of Transformer-based architectures for semantic segmentation [7, 46]. However, these methods are still computationally demanding. Transformer backbones. ViT [6] is the first work to prove that a pure Transformer can achieve state-of-the-art performance in image classification. ViT treats each image as a sequence of tokens and ϚϧνυϝΠϯରԠ 4FH'PSNFSͷޮՌ
hf149
kiyohiro8
frandle256
skoyamalab
aranciap
wata909
trafficbrain
gpeyre
dmattsun
kurita
kentaitakura
aiueola
nttcom
thoeni
frogandcode
smashingmag
tmm1
mclloyd
bryan
trishagee
csswizardry
thekraken
myddelton
lara
akosma
![ػց֮ϩϘςΟΫεݚڀάϧʔϓ த෦େֶϩΰ த෦େֶϩΰ ڭत ౻٢߂ Hironobu Fujiyoshi E-mail: [email protected]](https://files.speakerdeck.com/presentations/6bf6f8c602d74dc3be19e6cf483f98e4/slide_55.jpg){kind=link}