FlexiViT Paper Discussion

Transcript

1. Paper Discussion 26.01.2023 FlexiViT One Model for All Patch Sizes
2. The Tale of Sir FlexiViT
3. This is the tale of sir FlexiViT, Who struck all monsters as he saw fit. Our story begins in an old castle deep In the land where the stones lie asleep. The old stronghold stood mighty and tall, Loud and clear rang the angry king’s call.
4. “I’m loosing my treasure, I’m loosing my land. All of you fools sit waiting for gold in your hand.” He roared angry at the knights in the round, In all of his kingdom none better were found. All different and still all the same, All masters of their professional game.
5. Monsters to slay was their daily métier. In all sizes and colors they came every day. Monster so brutish and big as they’re dumb. Others small, canny, not stupid, not plumb. The knights they matched them so very alike, Some slower and rough, some quicker to strike.
6. No big knight could strike a small, cheeky imp. No small knight withstood a dragon’s big wing. Thus the king had hired a horde of the best. Each single had passed their personal test. But now he laments as the knights are so many. And costs they keep rising, he counts every Penny.
7. The same day a stranger knocked at the gate. Ready to lift the king’s grimmest fate. No sword, no armor, no gold in his hand. He looked like a peasant from the innermost land. “Hand me a weapon and a knight worthy horse So I can slay monsters from south to the north.”
8. The king devastated and in great despair. Gave him a blunt sword and a scraggy old mare. Months they had passed, they all thought him dead. When over the wall came a dead dragon’s head. An imp, a troll’s eye, a serpent’s long tail. Nobody thought that the man would prevail.
9. Yet there he stood, all covered in blood. Dirt in his hair. Pants, shoes full of mud. “I slayed them all for you, in different size. I care not their species, now I want my prize.” The king was delighted, invited him in. Announced him a knight for his special kin.
10. From this day on he was known as Sir FlexiViT, who struck monsters all sizes as he saw fit. All the other knights they were let go, The king cast them out without much ado. If you face a monster tall as a mountain or small as grass, Fear thy not for Sir FlexiViT will save you at last.
12. Recap: ViT Linear Projection * 0 1 2 3 4 Transformer Encoder MLP Head Class
13. Patch Size • More Patches = More Accuracy • More Patches = More Compute needed • Self-Attention Scales with O(s²) with s = total number of patches • Can be varied (e.g. Pre-training 8, finetune 4)
14. Patch Size Problem • Performance decreases when changing patch size WHY?
15. (Crazy) Math Part x = Patch (shape of pxp) w = Patch Embedding weights (pxp) ⟨x, w⟩ = Dot Product of x and w resize*p(y) = resize y to *p Problem: • ⟨x, w⟩ ≈ ¼⟨ resize2p(x), resize2p(w) ⟩ • Are there better ways to resize?
16. (Crazy) Math Part ŵ = new patch embedding weights B = Resize matrix • Optimize ŵ ∈ arg min ŵ 𝔼 x~X [(⟨x, w⟩-⟨Bx, ŵ⟩)²] • Solution is PseudoInverse (BT)+ PI-Resize
17. Pseudo Inverse Resize
18. FlexiViT
19. FlexiViT - Random Patch Size Batch 1 Batch 2 Batch 3 Batch 4
20. FlexiViT - Pseudo-Code
21. Knowledge Distillation ViT-B/6 FlexiViT-B Classify Horse Knight
22. Knowledge Distillation • Resize teacher’s patch and position embeddings • FunMatch Paradigm: ◦ Patient ◦ Consistent
23. Internal Representation
24. Transfer to Downstream
25. Finetuning
26. Flexify the Inflexible
27. Ablations - Other Resize Options
28. Ablations - Fixed Shape Size
29. THANK YOU
30. Knight Helmet on cover by brgfx on Freepik Sword on cover by catalysstu on Freepik All images that are not part of the original paper were created using Dreamstudio and StableDiusion 1.5/2.1 The Slide Theme is called “The Feast of St. Louis” and can be found on slidesgo: hps://slidesgo.com/new-theme/feast-of-st-louis-minitheme Other Resources: Google/BigVision PR “Open-sourcing FlexiViT” Credits