Mean Field Games with Applications 1

1 !2

Work of 2018-2019 Social Network Engineering Gradient flow !3

Mean-field games (MFG) framework Introduced by P.-L. Lions, J.-M. Lasry in the mathematics community and M. Huang, P. Caines, R. Malham´ e in the engineering one to model huge populations of identical, rational agents that play non-cooperative di↵erential games. In a PDE form, MFG resemble macroscopic equations from statistical physics in game-theoretic/optimal control framework. Yield near optimal solutions for huge populations of optimizing agents. Thus, tremendous simplification in practical problems. engineer field !4

Nash equilibrium in a differential game A continuum of players with a distribution density ⇢(x, t) play a non-cooperative di↵erential game. An individual agent faces an optimization problem (x, t) = inf E t Z 0 L(Xs, vs, ⇢(·, s), s)ds + 0(X0, ⇢(·, 0)), (9) where dXs = vsds + p 2 dWs, XT = x, and L is the Lagrangian. In Nash equilibrium, each agent cannot do better unilaterally. Optimal strategies are given by the value function, , that solves the corresponding Hamilton-Jacobi equation. The density, ⇢(x, t), evolves according to optimal actions of agents. Thus, the Fokker-Planck equation. !5

The PDE formulation A typical time-dependent MFG system (derived from (9)) has the form 8 > > > > > < > > > > > : t 2 + H (x, Dx (x, t), ⇢(x, t), t) = 0 ⇢t 2 ⇢ divx (⇢DpH (x, Dx (x, t), ⇢(x, t), t)) = 0 ⇢ 0, R Td ⇢(x, t)dx = 1, t 2 [0, T] ⇢(x, T) = ⇢T (x), (x, 0) = 0(x, ⇢(x, 0)), x 2 Td . (10) Given: H : Td ⇥ Rd ⇥ X ⇥ R ! R is a periodic in space and convex in momentum Hamiltonian, where X = R+ or X = L 1(Td ; R+) or X = R+ ⇥ L 1(Td ; R+). H = L ⇤ is a di↵usion parameter 0 : Td ⇥ X ! R, ⇢T : Td ! R given initial-terminal conditions Unknowns: , ⇢ : Td ⇥ [0, T] ! R above !6

Classification of MFG Mean-field interaction or non-linearity is the dependence of H, L on the density, ⇢. Local interaction if H, L depend on ⇢ locally. For instance, H(x, p, ⇢, t) = ˜ H(x, p, t) ln ⇢ H(x, p, ⇢, t) = ˜ H(x, p, t) ⇢ Non-local interaction if H, L depend on ⇢ non-locally. For instance, H(x, p, ⇢, t) = ˜ H(x, p, t) F 0 @ x, Z Td K(x, y)⇢(y)dy 1 A Congestion interaction if we penalize moving in dense area. For instance (soft congestion) H(x, p, ⇢, t) = ⇢↵ ˜ H ✓ x, p ⇢↵ , ⇢, t ◆ , ↵ > 0 !7

U(t, µ) = inf ⇢2P(Td) F(⇢) + dW (⇢, µ)2 2t Hopf-Lax on density space E.g. (Burgers’) Hamilton-Jacobi on density space Characteristics on density space (Nash equilibrium in mean field games) @tµs + r · (µs r s) = 0 @t s + 1 2 (r s)2 = 0 Math Review: mean field games @ @t U(t, µ) + 1 2 Z Td (r µ U(t, µ))2µ(x)dx = 0, U(0, µ) = F(µ) !8

Density Control: Optimal Transport dW (⇢, µ)2 = inf ⇢s,us n Z 1 0 Z R u2 s ⇢sdxdt: @s⇢s + r · (⇢sus) = 0, ⇢0 = ⇢, ⇢1 = µ o 8 What happens if the total population density is not normalized/balanced? 9

works !10

Change delta F to negative delta F mean field game !11

| Z µ Udx|2 !12

!13

0 1 0 Density (7(x; t)) Time (t) 0.5 0.5 0.5 Space (x) 1 0 1 14

Unnormalized L1 Wasserstein metric Let p = 1: UW1(µ0, µ1) = inf v,f(t) n Z 1 0 Z ⌦ kvkµdxdt + 1 ↵ Z 1 0 |f(t)|dt · |⌦|: @tµ + r · (µv) = f(t) o . 15 15

Time independent solution Denote m(x) = Z 1 0 v(t, x)µ(t, x)dt, with the fact Z 1 0 f(t)dt = c = 1 |⌦| ⇣ Z ⌦ µ1(x)dx Z ⌦ µ0(x)dx ⌘ . then by Jensen’s inequality and integrating the time variable t, we obtain UW1(µ0, µ1) = inf m n Z ⌦ km(x)kdx + 1 ↵ Z ⌦ µ0(x)dx Z ⌦ µ1(x)dx : µ1(x) µ0(x) + r · m(x) = 1 |⌦| ⇣Z ⌦ µ1(x)dx Z ⌦ µ0(x)dx ⌘o . 16 From this equation, we see the momentum is independent of alpha. 16

Closed form solution In one space dimension on the interval ⌦ = [0, 1], the L1 unnormalized Wasserstein metric has the following explicit solution: UW1(µ0, µ1) = Z ⌦ Z x 0 µ1(y)dy Z x 0 µ0(y)dy x Z ⌦ (µ1(z) µ0(z))dz dx + 1 ↵ ⇣ Z ⌦ µ1(z)dz Z ⌦ µ0(z)dz ⌘ . 17 If the total masses are equal, the additional term disappears. 17

Unnormalized L2 Wasserstein metric Let p = 2: UW2(µ0, µ1) = inf v,f(t) n Z 1 0 Z ⌦ kvk2µdxdt + 1 ↵ Z 1 0 f(t) 2dt · |⌦|: @tµ + r · (µv) = f(t) o . 22 18

Minimizer system The minimizer (v(t, x), µ(t, x), f(t)) for UOT problem satisfies v(t, x) = r (t, x), f(t) = ↵ 1 |⌦| Z ⌦ (t, x)dx, and 8 > > > > < > > > > : @tµ(t, x) + r · (µ(t, x)r (t, x)) = ↵ 1 |⌦| Z ⌦ (t, x)dx @t (t, x) + 1 2 kr (t, x)k2 = 0 µ(0, x) = µ0(x), µ(1, x) = µ1(x). 23 19

Unnormalized Monge-Ampere equation Denote (x) = 1 2 kxk2 + (0, x), Following the Hopf-Lax formula, the minimizer of unnormalized OT satisfies µ(1, r (x))Det(r2 (x)) µ(0, x) =↵ Z 1 0 Det ⇣ tr2 (x) + (1 t)I ⌘ · Z ⌦ ⇣ (y) kyk2 2 + tkr (y) yk2 2 ⌘ Det ⇣ tr2 (y) + (1 t)I ⌘ dydt. 24 20

Unnormalized Kantorovich problem 1 2 UW2(µ0, µ1) 2 = sup n Z ⌦ (1, x)µ(1, x)dx Z ⌦ (0, x)µ(0, x)dx ↵ 2 Z 1 0 ⇣ Z ⌦ (t, x)dx ⌘2 dt o where the supremum is taken among all : [0, 1] ! ⌦ satisfying @t (t, x) + 1 2 kr (t, x)k2  0. 25 This is connected to one of UCLA’s fast algorithms for OT by M. Jacobs and F. Leger, 2019. 21

Primal-dual method !22

Algorithm for MFG !23

Example I 29 24 OT

Example II 30 25

Example 31 26

Generalized unnormalized OT In fact, when p=2, the minimization has the unconstrained optimization formulation: Models suggested by Chen, Georgiou, Ling, Tannenbaum, 2017 for p=1,2. where Lµ = r · (µr) !27

Unconstrained formulation The corresponding gradient descent flow becomes !28

Gradient flow algorithm !29

Examples of unnormalized OT in 1D f(t,x) f(t) !30

Examples of unnormalized OT in 2D f(t,x) !31

Mean field game on manifolds Here we replace the domain of MFG from torus strategy set to a manifold M, point clouds, and graphs (discrete strategy set) U(t, µ) = inf v,µ,f Z 1 0 Z M kv(t, x)kpµ(t, x) F(µ(t, ·))dxdt + G(µ(0, ·)) such that the dynamical constraint is satisfied @tµ + r · (µv) = 0, µ(1, x) = µ1(x) Consider !32

Optimal transport on Manifold !33

MFG in 1D via unconstrained optimization Evolution of density with Phi(0,x) fixed !34

Classical MFG examples via unconstrained optimization The evolution of density Quadratic interaction energy with linear periodic potential energy We test our algorithms using examples in Achdou, Capuzzo-Dolcetta, Mean Field Games: Numerical Methods, 2009. !35

Classical MFG examples via unconstrained optimization The evolution of density: quadratic interaction energy with linear periodic potential energy !36

Spectral methods in nonlocal MFG Goal: Solve MFG with nonlocal interactions 8 > < > : t + D + H(x, r ) = R ⌦ K(x, y)⇢(y, t)dy, ⇢t + D⇤⇢ div(⇢rpH(x, r )) = 0, (x, 0) = 0(x), ⇢(x, T) = ⇢T (x). (13) Above, K is a monotone symmetric interaction kernel, D is a monotone linear operator (e.g. D = ). Strategy: Model the interaction in a Fourier space. Specifically, given a basis {x 7! fi (x)}i expand the kernel K(x, y) = X i,j kij fi (x)fj (y). !37

Furthermore, Z ⌦ K(x, y)⇢(y, t)dy = X i ai (t)fi (x), where ai (t) = X j kij Z ⌦ ⇢(y, t)fi (y)dy. Hence, (13) can be written as 8 > > > > < > > > > : t + D + H(x, r ) = P i ai (t)fi (x), ⇢t + D⇤⇢ div(⇢rpH(x, r )) = 0, ai (t) = P j kij R ⌦ ⇢(y, t)fi (y)dy, 8i, (x, 0) = 0(x), ⇢(x, T) = ⇢T (x). (14) The problem !38

New variational principle: System (14) is equivalent to inf a ⇢ a · K 1 a 2 Z ⌦ a(x, T)⇢T (x)dx s.t. (16) holds , (15) where K = (kij ), and a is the viscosity solution of ( t + D + H(x, r ) = P i ai (t)fi (x), (x, 0) = 0(x). (16) Advantages: (15) is a convex optimization problem! Flexible and sparse representations of mean-field interactions Grid-free approximation methods for first-order problems Compatibility with powerful convex optimization techniques such as ADMM, PDHG the system It is !39

Examples: Take a periodic domain ⌦ = T2, and periodic Gaussian interaction kernels K ,µ (x, y) = 2µ2 ⇡ 2 X ↵2Z2 e 2|x y ↵|2 2 , x, y 2 T2. (17) Furthermore, suppose that H(x, p) = |p|2 2 , (x, p) 2 T2 ⇥ R2, D = 0. (18) Observation: Trigonometric system yields diagonal (sparse) representations for translation invariant kernels! !40

Figure: Gaussian kernels for ( , µ) = (0.1, 0.75), (0.1, 0.5), (1, 0.5). Figure: The terminal distribution ⇢T (x1, x2 ) and the initial cost 0 (x1, x2 ). !41

!42

!43

!44

Numerical method 2D !45

!46

!47

Constrained MFG Given a parameterized density ⇢(✓, x) , consider the variational problem @t⇢(✓(t), x) + r · (⇢(✓(t), x)v(t, x)) = 0, ✓(T) fixed such that the constrained continuity equation holds Typical parameterized density are given by probability models, such as mixed Gaussian, deep neural networks, graphical models etc. inf ⇢(✓,·) Z T 0 n Z L(x, v)⇢(✓(t), x)dx F(⇢(✓(t), ·)) o dt + G(⇢(✓(0), ·)) !48

submanifold !49

Evolutionary dynamics for Robotics min ⇢ KL(⇢k⇢ref ) + 1 2 Z Z W(x, y)⇢(x)⇢(y)dxdy Wasserstein Gradient flow @t⇢ = r · (⇢rW ⇤ ⇢) + r · (⇢r log ⇢ ⇢ref ) Lagrangian coordinates ˙ X = rW ⇤ ⇢ + r log ⇢ ⇢ref !50

!51

!52

Examples !53