Mean Field Game MURI Report Stanley Osher (UCLA) August, 2021 1 AFOSR MURI FA9550-18-1-0502

MURI’s Product 2

5G communication AI UAV COVID 19 pandemic control 3

Mean field game 4

Mean field system A typical time-dependent MFG system (derived from (9)) has the form 8 > > > > > < > > > > > : t 2 + H (x, Dx (x, t), ⇢(x, t), t) = 0 ⇢t 2 ⇢ divx (⇢DpH (x, Dx (x, t), ⇢(x, t), t)) = 0 ⇢ 0, R Td ⇢(x, t)dx = 1, t 2 [0, T] ⇢(x, T) = ⇢T (x), (x, 0) = 0(x, ⇢(x, 0)), x 2 Td . (10) Given: H : Td ⇥ Rd ⇥ X ⇥ R ! R is a periodic in space and convex in momentum Hamiltonian, where X = R+ or X = L 1(Td ; R+) or X = R+ ⇥ L 1(Td ; R+). H = L ⇤ is a di↵usion parameter 0 : Td ⇥ X ! R, ⇢T : Td ! R given initial-terminal conditions Unknowns: , ⇢ : Td ⇥ [0, T] ! R has the form 5

Mean Field Control Problems For Vaccine Distribution Wonjun Lee, Siting Liu, Wuchen Li, Stanley Osher 2021 1 6

2 7

Vaccine: Manufacturing, Delivery, and Consumption When and where to deliver? I Amount of vaccine is limited. I Vaccine production is limited I Delivery is expensive 3 8

Spatial SIRV dynamic model We formulate the optimization problem that satisfies the following equations and constraints. @t⇢S + r · mS = ⇢S K ⇤ ⇢I + ⌘2 S 2 ⇢S ✓1⇢V ⇢S @t⇢I + r · mI = ⇢S K ⇤ ⇢I ⇢I + ⌘2 I 2 ⇢I @t⇢R + r · mR = ⇢I + ⌘2 R 2 ⇢S + ✓1⇢V ⇢S PDE of the vaccine distribution @t⇢V = f(t, x) ✓1⇢V ⇢S , 0  t < T 0 @t⇢V + r · mV = ✓2⇢V ⇢S , T 0  t  T and constraints on the vaccine density function ⇢V and the factory function f(t, x). 0  f(t, x)  fmax (t, x) 2 [0, T 0] ⇥ ⌦factory f(t, x) = 0 (t, x) 2 [0, T 0] ⇥ ⌦\⌦factory ⇢V (t, x)  Cfactory (t, x) 2 [0, T 0] ⇥ ⌦factory 4 Vaccine model 9

Methods Consider X i=S,I,R,V Ei(⇢i(T, ·)) + Z T 0 Z ⌦ X i=S,I,R Fi(⇢i, mi)dx dt + Z T T 0 Z ⌦ FV (⇢V , mV )dx dt + Z T 0 GP ((⇢S + ⇢I + ⇢R)(t, ·)) + GV (⇢V (t, ·))dt + Z T 0 0 G0(f(t, ·))dt Here, the cost functional satisfies the following conditions. I minimize the transportation cost for moving each population; I minimize the total number of infected people and susceptible people by maximizing the usage of the vaccines at time T ; I maximize the total number of recovered people at time T ; I avoid high concentration of population and vaccines at each time t 2 (0, T ); I minimize the amount of vaccines produced during t 2 (0, T 0); I minimize the transportation cost for delivering vaccines during t 2 (T 0, T ). 5 10

Mean field systems for Vaccine distributions We highlight the dynamical systems of vaccine distributions in the model. 8 > > > > > > > > > < > > > > > > > > > : @t V + GV ⇢ (⇢V ) + ⇢S ✓1( R S) ✓2 V ) = 0 @t V ↵V 2 |r V |2 + GV ⇢ (⇢V ) + ⇢S ✓1( R S) ✓2 V ) = 0 @t⇢V f + ✓2⇢S⇢V = 0 @t⇢V 1 ↵V r · (⇢V r V ) + ✓2⇢S⇢V = 0 6 11

Example I 7 12

Example II 8 13

Future directions The proposed mean field control formulations can be adjusted to model other natural disasters. I Rescue/Control problems in California wildfires; I Evaluation plans for Florida hurricanes, e.t.c. References: I Lee, W., Liu, S., Li, W., Osher, S. (2021). Mean field control problems for vaccine distribution. arXiv:2104.11887. 9 Evacuation 14

A Fast Proximal Gradient Method and Convergence Analysis for Dynamic Mean Field Planning Jiajia Yu1 Rongjie Lai1 Wuchen Li2 Stanley Osher3 June 7, 2021 1Department of Mathematics, Rensselaer Polytechnic Institute, United States. 2Department of Mathematics, University of South Carolina, United States. 3Department of Mathematics, University of California, Los Angeles, United States. Funding: J. Yu and R. Lai’s work are supported in part by an NSF career award DMS–1752934. W. Li and S. Osher’s work are supported in part by AFOSR MURI FP 9550-18-1-502. 15

Introduction Density transporting/fitting (a) crowd simulation (b) source distribution (c) generative models Applications of density transporting/fitting 2 16

Formulations Let C(⇢0, ⇢1) := {(⇢, m) : @t⇢ + rx · m = 0, ⇢(0, ·) = ⇢0, ⇢(1, ·) = ⇢1} . I Optimal Transport min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) dxdt. I Mean Field Planning min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + F(x, ⇢(t, x))dxdt. Let C(⇢0) := {(⇢, m) : @t⇢ + rx · m = 0, ⇢(0, ·) = ⇢0}. I Mean Field Game min ⇢,m2C(⇢0) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + F(x, ⇢(t, x))dxdt + Z ⌦ G(x, ⇢(1, x))dx. 3 17

Algorithm: Proximal Gradient Descent MFP: min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + F(x, ⇢(t, x))dxdt | {z } Y(⇢,m) , min ⇢,m Y(⇢, m) + XC(⇢0,⇢1) (⇢, m) where XC(⇢0,⇢1) (⇢, m) = ( +1, (⇢, m) 62 C(⇢0, ⇢1), 0, (⇢, m) 2 C(⇢0, ⇢1). Proximal Gradient Descent method for MFP: 8 > > > > > > > > > > > > > > < > > > > > > > > > > > > > > : ⇣ ⇢k+ 1 2 , mk+ 1 2 ⌘ = ⇢k, mk ⌘k ⇢,mY ⇢k, mk , | {z } gradient (forward) descent easy and fast to compute for di↵erentiable F ⇢k+1, mk+1 = ProjC(⇢0,⇢1) ⇣ ⇢k+ 1 2 , mk+ 1 2 ⌘ , | {z } proximal (backward) descent = ⇣ ⇢k+ 1 2 , mk+ 1 2 ⌘ rt,m 1 t,mrt,m · ⇣ ⇢k+ 1 2 , mk+ 1 2 ⌘ . | {z } independent to the form of F fast to compute with Fast Cosine Transformation 4 18

Summary: E ciency and Flexibility PGD is a flexible algorithm for MFP: I The gradient descent step is easy to compute as long as regularizer F is di↵erentiable. I The proximal descent step is independent to the form of F. PGD is also e cient: I The gradient descent step can be implemented elementwise. I The proximal descent step is e cient to compute with fast cosine transform. I If ⌦ = [0, 1]d 1 and the domain [0, 1] ⇥ ⌦ is uniformly discretized to nd grids, then the computation cost for each step is O(dnd log n). I By applying accelerated PGD and multigrid strategies, we can further reduce the number of iterations and total time for convergence. 5 19

Convergence: Discretized solution ! continuous optimizer Our algorithm produces (Pk, Mk) as an approximation of (⇢k, mk) on grid points. With mild assumptions, We, for the first time, establish the following convergence based on an optimization algorithm: Theorem (Yu-Lai-Li-Osher’21) I (Pk, Mk) ! (⇢k, mk) on grid points for any fixed k as mesh size h ! 0. I (Pk, Mk) ! (⇢⇤, m⇤) on grid points as h ! 0, k ! 1. I (P⇤, M⇤) ! (⇢⇤, m⇤) on grid points as h ! 0 6 20

E ciency and Accuracy (Optimal Transport) min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) dxdt Table: E ciency and accuracy comparison of OT with best performance highlighted in red. (ALG=Augmented Lagrangian, G-prox=G-prox PDHG. Left: 1D. Right: 2D. nt =64, nx = ny =256.) 7 21

Flexibility I: regions with obstacles min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + Q⇢(t, x)Q(x)dxdt where Q(x) = ( 1, x 2 white region, 0, otherwise , . Figure: Evolution of densities for various ⇢0, ⇢1, Q(x). 8 22

Flexibility II: di↵erent density regularizers min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + Q⇢(t, x)Q(x) + EFE(⇢(t, x))dxdt where Q(x) = ( 1, ⇢0(x) = 0 and ⇢1(x) = 0, 0, otherwise , . (a) Q = 0, FE(a) = 0 (b) Q 6= 0, FE(a) = a2 2 (c) Q 6= 0, FE(a) = 1 a Figure: Evolution of densities for MFP with di↵erent density regularizers. 9 23

Generalization to MFG (MFP) min ⇢,m2C(⇢0,⇢1) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + F(x, ⇢(t, x))dxdt (MFG) min ⇢,m2C(⇢0) Z 1 0 Z ⌦ km(t, x)k2 2 2⇢(t, x) + F(x, ⇢(t, x))dxdt Z ⌦ ⇢1(x)⇢(1, x)dx (a) MFP (b) MFG Figure: Evolution of densities for OT, MFP and MFG. 10 Mean field planning Mean field Control/Game 24

APAC-Net: Alternating the Population and Control Neural Networks, for Mean-Field Games Problems Alex Tong Lin Joint with: Samy Wu Fung, Levon Nurbekyan, Wuchen Li, and Stanley Osher June 5, 2021 1 25

Variational Mean-Field Games I Namely for some mean-field games, the equilibrium solution can be found by minimizing an overall “energy” (e.g. multiply the value function for a single agent by ⇢): A(⇢, v) = min ⇢,v Z T 0 Z ⌦ ⇢(x, t)L(v) + F(x, ⇢) dx dt + Z ⌦ g(x)⇢(x, T) dx where x = x(t), v = v(t) above, and where the optimization has the constraint, @t⇢ ⌫ ⇢ + div(⇢v) = 0. 2 26

Variational Mean-Field Games / APAC-Net Preview After some elevating the constraints into the objective, integrating by parts, and calculating the Legendre transform, we get = max ' min ⇢ Z T 0 Z ⌦ F(x, ⇢(x, t)) dx dt + Z ⌦ ⇣ g(x) '(x, T) ⌘ ⇢(x, T) dx + Z ⌦ '(x, 0)⇢(x, 0) dx + Z T 0 Z ⌦ ✓ @t'(x, t) + '(x, t) L⇤( r'(x, t)) ◆ ⇢(x, t) dx dt which means we end up with a sampling problem – this is a preview of APAC-Net. This is in the spirit of Feynman-Kac. Then the idea is to turn ⇢ and ' into neural networks and train as in GANs (Generative Adversarial Networks). 3 27

GAN Training I Training has a discriminator and a generator. The generator produces samples (analogous to our ⇢), and the discriminator evaluates the quality of those samples (analogous to our '). 4 28

Why Neural Networks? Because the usual way to do things – using a grid to discretize space – becomes impossible in high dimensions. 5 29

Numerical Results - Obstacles & Congestion I H(p) = kpk2 I A 20 dimensional obstacle problem where we have an obstacle in (x1, x2) and in (x3, x4) and in (x5, x6), etc. Congestion penalty is active in the bottlenecks. Figure: A screencapture of a video that will be played 6 30

Numerical Results: A realistic example, the Quadcopter The dynamics of a quadcopter are: 8 > > > > > > < > > > > > > : ¨ x = u m (sin( ) sin( ) + cos( ) cos( ) sin(✓)) ¨ y = u m ( cos( ) sin( ) + cos( ) sin(✓) sin( )) ¨ z = u m cos(✓) cos( ) g ¨ = ˜ ⌧ ¨ ✓ = ˜ ⌧✓ ¨ = ˜ ⌧ where u is the thrust, g is the gravitational acceleration (9.81m/s2), and x, y, z are the spatial coordinates, , ✓, are the angular coordinates, and ˜ ⌧ , ˜ ⌧✓ , ˜ ⌧ . Turns 12-dimensional when you transfer to first-order system. 7 31

Numerical Results: A realistic example, the Quadcopter 8 32

Numerical Results: A realistic example, the Quadcopter Movie: Figure: A screencapture of a video that will be played 9 33