Continuous Depth models and a brief overview of Neural ODE

Foundation of Countinuous Depth ODE, Interpretation & Applications
Foundation of continuous-depth models
.. and a brief introduction to Neural ODE
Ayan Das
PhD Student
SketchX, CVSSP
SketchX meetup talk
September 2, 2021
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 1 / 15

View Slide

Presentation Overview
1 Foundation of Countinuous Depth
2 ODE, Interpretation & Applications

View Slide

Foundation of Countinuous Depth
Motivation
Revisiting ResNet
Modifying ResNet

View Slide

Motivation
Deep architectures so far
Deep models so far
1 Notion of depth is discrete

View Slide

Motivation
Deep models so far
2 Parameter count increases with number of layers

View Slide

Motivation
Deep models so far
2 Parameter count increases with number of layers
3 Backpropagation takes O(L) memory

View Slide

Motivation
Design Motivation
Can we have ?
1 Continuous notion of Depth

View Slide

Motivation
Design Motivation
Can we have ?
2 Input “evolves” along depth

View Slide

Motivation
Design Motivation
Can we have ?
2 Input “evolves” along depth
3 Low cost Backpropagation

View Slide

Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)

View Slide

Revisiting ResNet
Generic ResNet
For each layer l
xl+1 ← xl + g(xl)

View Slide

Revisiting ResNet
Generic ResNet
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···

View Slide

Revisiting ResNet
Generic ResNet
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···
ResNet architecture (no downstream task)
i.e., several blocks cascaded
xl+1 ← xl + g(xl )
xl+2 ← xl+1 + g(xl+1 )
xl+3 ← xl+2 + g(xl+2 )

View Slide

Revisiting ResNet
Generic ResNet
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···
ResNet architecture (no downstream task)
i.e., several blocks cascaded
xl+1 ← xl + g(xl; Θ1)
xl+2 ← xl+1 + g(xl+1; Θ2)
xl+3 ← xl+2 + g(xl+2; Θ3)
Remeber !
They have different parameters, where majority of the
ResNet’s modelling capacity lies

View Slide

Modifying ResNet
Exploiting ResNet structure
.. for countinuous depth models
Precisely, three changes .. x1 ← x0 + g(x0; Θ1)
x2 ← x1 + g(x1; Θ2)
x3 ← x2 + g(x2; Θ3)

View Slide

Modifying ResNet
Precisely, three changes ..
1 Scale g(·) with a scalar ∆l
→ Quite harmless, g(·) can always adjust by learning to
scale it up
x1 ← x0 + g(x0; Θ1)·∆l
x2 ← x1 + g(x1; Θ2)·∆l
x3 ← x2 + g(x2; Θ3)·∆l

View Slide

Modifying ResNet
scale it up
2 Share parameters accross blocks
→ Reduces number of parameters, hence modelling
capacity
x1 ← x0 + g(x0; Θ)·∆l
x2 ← x1 + g(x1; Θ)·∆l
x3 ← x2 + g(x2; Θ)·∆l

View Slide

Modifying ResNet
scale it up
2 Share parameters accross blocks
→ Reduces number of parameters, hence modelling
capacity
3 much more blocks than usual
→ To compensate model capacity, we want more blocks
x1 ← x0 + g(x0; Θ)·∆l
x2 ← x1 + g(x1; Θ)·∆l
x3 ← x2 + g(x2; Θ)·∆l
.
.
.
x4 ← x3 + g(x3; Θ) · ∆l
x5 ← x4 + g(x4; Θ) · ∆l
x6 ← x5 + g(x5; Θ) · ∆l
.
.
.
xL ← xL−1 + g(xL−1; Θ) · ∆l

View Slide

Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 + g(x0; Θ) · ∆l + g(x1; Θ) · ∆l + · · · + g(xL−1; Θ) · ∆l

View Slide

Modifying ResNet
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l

View Slide

Modifying ResNet
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
Can we have infinitely long ResNet ?
i.e., L → ∞

View Slide

Modifying ResNet
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
i.e., L → ∞
The summation might blow up (in either direction)
−∞ ← xL → +∞

View Slide

Modifying ResNet
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
i.e., L → ∞
The summation might blow up (in either direction)
−∞ ← xL → +∞
Solution
Simulteneously do the following:
L → ∞
∆l → 0

View Slide

Modifying ResNet
Emergence of Continuous Depth
In limiting case:
xL = x0+ lim
L→∞
∆l→0
L−1
l=0
g(xl; Θ) · ∆l

View Slide

Modifying ResNet
Emergence of Continuous Depth
In limiting case:
xL = x0+ lim
L→∞
∆l→0
L−1
l=0
g(xl; Θ) · ∆l
xL = x0+
L
0
g(xl; Θ)dl

View Slide

Modifying ResNet
Continuous Depth ResNet
A new kind of ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y

View Slide

Modifying ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Parameterized by two things
1 Internal function g, along with Θ
2 Integration limits, i.e.
l2
l1

View Slide

Modifying ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Parameterized by two things
1 Internal function g, along with Θ
2 Integration limits, i.e.
l2
l1
Surprised ?
L can be fraction. So, L = 1.234 layer network (?)

View Slide

Modifying ResNet
Continuous Depth RNN
Intermediate values matter

View Slide

Modifying ResNet
x +
l1 or l2 or l3
0
g(x; Θ)dl

View Slide

Modifying ResNet
x +
l1 or l2 or l3
0
g(x; Θ)dl
Unlike RNN, we can produce intermediate values at any (non-uniform) interval

View Slide

ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Interpretation
Potential applications
Practical Usage

View Slide

Where is the Differential Equation?
Neural Ordinary Differential Equation1
y = x +
L
0
g(x; Θ)dl
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)

View Slide

y = F(l = 0) +
L
0
g(x; Θ)dl

View Slide

F(l = L) = F(l = 0) +
L
0
g(x; Θ)dl

View Slide

F(l = L) = F(l = 0) +
L
0
∂F
∂l
dl
A popular ODE solver called Eular Integration

View Slide

F(l = L) = F(l = 0) +
L
0
∂F
∂l
dl
A popular ODE solver called Eular Integration
Proposed first in NeurIPS 2018 (best) paper titled ”Neural ODE”1.
Parameterize the derivative as a Neural Network:
∂F
∂l
= NeuralNet(x; Θ)

View Slide

Backpropagation
Naive Backpropagation
How to we backpropagate ?
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y

View Slide

Backpropagation
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y
Why not just discretize and backpropagate trivially ?

View Slide

Backpropagation
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y
Why not just discretize and backpropagate trivially ?
Nope, don’t do it!
1 Introduces discretization error
2 Memory consumption blows up

View Slide

Backpropagation
Adjoint Backpropagation12
x → x +
L
0
∂F
∂l
dl → y
∂(loss)
∂x
,
∂(loss)
∂Θ
← Adjoint Backprop. ←
∂(loss)
∂y
2https://ayandas.me/blog-tut/2020/03/20/neural-ode.html

View Slide

Interpretation
Standard Interpretation
.. from calculus
Initial Value Problem
1
∂F
∂l
= NeuralNet(x; Θ) is a vector field

View Slide

Interpretation
.. from calculus
1
∂F
∂l
2 x → y is its solution trajectory by solving the IVP

View Slide

Interpretation
.. from calculus
1
∂F
∂l
Fitting
By changing Θ, we can fit a given x0 & Y

View Slide

Interpretation
.. from calculus
1
∂F
∂l
Fitting
By changing Θ, we can fit a given x0 & Y
Supervise intermediate values to fit a whole
trajectory

View Slide

Potential applications
Lifespan synthesis1
A possible appraoch
1https://grail.cs.washington.edu/projects/lifespan age transformation synthesis/

View Slide

Practical Usage
Practical Implementation (torchdiffeq1)
.. in PyTorch code
# libraries like ’torchdiffeq’
from torchdiffeq import odeint_adjoint
# The dF/dl = g(x; theta)
class Dynamics(nn.Module):
def forward(self, x):
return ...
# forward pass of the ODE layer
y = odeint_adjoint(Dynamics(), x0, [0., 1.])
1https://github.com/rtqichen/torchdiffeq

View Slide

End of Presentation
Thank You
ayandas.me dasayan05 dasayan05

View Slide

Continuous Depth models and a brief overview of Neural ODE

Continuous Depth models and a brief overview of Neural ODE

Ayan Das

More Decks by Ayan Das

Other Decks in Research

Featured

Transcript