NFNet: High-Performance Large-Scale Image Recognition Without Normalization

This is a talk about SAM, image augmentation, ResNet-like architectures evolution (from ResNet to NFNet), NAS (neural architecture search), and other techniques which could help to build modern SOTA in Computer Vision and Object Detection.

Transcript

1. NFNet: High-Performance Large-Scale Image Recognition Without Normalization Alexey Zinoviev, JetBrains
2. Bio • Java & Kotlin developer • Distributed ML enthusiast • Apache Ignite PMC • TensorFlow Contributor • ML engineer at JetBrains • Happy father and husband • https://github.com/zaleslaw • https://twitter.com/zaleslaw
3. NFNets: top-1 accuracy vs training latency
4. Evolution of Image Recognition models
5. BatchNorm + Skip Connections Epoch NFNet-F4+
6. Top-1 Accuracy 2021
7. Architecture innovations: 2015-2019
8. Residual Block
9. Bottleneck Residual Block
10. Bottleneck Residual Block
11. Depthwise Separable Convolution
12. Inception Module
13. Multi-residual block: ResNeXt
14. Shared source skip connections
15. Dense blocks in DenseNet (DenseNet != MLP)
16. AutoML Era
17. New principle in AutoML era Give us your computational resource limit and we scale our baseline model for you
18. EfficientNet: scale <width, depth, resolution>
19. 3D Pareto: FLOPS, Number of params and Top-1 Acc
20. HPO task
21. Bilevel optimization problem
22. NAS (neural architecture search)
23. Search Space skeleton
24. Controller implementation (RNN, isn’t it?)
25. NASNet cells designed by AutoML algorithm
26. Evolutionary algorithms
27. Evolutionary AutoML
28. Evolution vs RL vs Random Search
29. AmoebaNet: pinnacle of evolution
31. Batch Normalization
32. Internal Covariate Shift
33. Batch Norm algorithm
34. Bad parts • Batch normalization is expensive • Batch normalization breaks the assumption of data independence • Introduces a lot of extra hyper-parameters that need further fine-tuning • Causes a lot of implementation errors in distributed training • Requires a specific “training” and “inference” mode in frameworks
35. The philosophy of this paper Identify the origin of BatchNorm’s benefits and replicate these benefits in BatchNorm-free Neural Networks
36. Good parts • Batch normalization downscales the residual branch • Batch normalization eliminates mean-shift (in ReLU networks) • Batch normalization has a regularizing effect • Batch normalization allows efficient large-batch training
37. Early Free Batch Norm architects
38. Residual Branch Downscaling effect: SkipInit Batch Normalization Biases Deep Residual Networks Towards Shallow Paths
39. Removing Mean Shift: Scaled Weight Standardization Characterizing signal propagation to close the performance gap in unnormalized ResNets
40. NFNet improvements
41. Improved SkipInit
42. Changed Scaled Weight Standardization
43. Regularization: Dropout Dropout CNNs BatchNorm me
44. Regularization: Stochastic Depth
45. Regularization: Stochastic Depth
46. How to train on larger batches?
47. Gradient Clipping
48. Intuition about Gradient Clipping Parameter updates should be small relative to the magnitude of the weight
49. Adaptive Gradient Clipping
50. Not enough Top-1 Accuracy!!! Even with Adaptive Gradient Clipping, and the modified residual branch and convolutions, normalizer-free networks still could not surpass the accuracies of EfficientNet.
51. New SOTA model family
52. Architecture optimization for improved accuracy and training speed • SE-ResNeXt-D as a baseline • Fixed group width (specific for the ResNeXt architecture) • Depth Scaling pattern was changed (from very specific to very simple) • Width pattern was changed too
53. Bottleneck block design
56. The whole NFNet family
57. Implementation • On JAX from authors • Collab to play with • PyTorch with weights • Yet another PyTorch • Very good PyTorch (clear code) • Adaptive Gradient Clipping example • Broken Keras example (could be a good entry point) • Raw TF implementation
58. SAM: Sharpness Aware Minimization
59. SAM: Sharpness Aware Minimization
60. SAM: Two backprops and approximate grads
61. Accelerating Sharpness-Aware Minimization The idea is to use 20% of batch on SAM step only.
62. Modern Augmentation
63. Different augmentations applied to Baseline
64. RandAugment
65. CutMix, MixUp, Cutout
66. What model is the best?
68. BN vs NF for different depths and resolutions
69. Summary 1. Downscales residual branch 1. NF-strategy
70. Summary 1. Downscales residual branch 2. Enables large batch training 1. NF-strategy 2. Adaptive gradient clipping
71. Summary 1. Downscales residual branch 2. Enables large batch training 3. Implicit regularization 1. NF-strategy 2. Adaptive gradient clipping 3. Explicit regularization
72. Summary 1. Downscales residual branch 2. Enables large batch training 3. Implicit regularization 4. Prevents mean-shift 1. NF-strategy 2. Adaptive gradient clipping 3. Explicit regularization 4. Scaled Weight Standardization
73. Summary 1. Downscales residual branch 2. Enables large batch training 3. Implicit regularization 4. Prevents mean-shift 1. NF-strategy 2. Adaptive gradient clipping 3. Explicit regularization 4. Scaled Weight Standardization • NFNets were new SOTA during a few months • NFResNet >= BNResNet • NFNet training >>> EfficientNet training