Wuchen Li
August 09, 2019
120

# Mean field games with applications

August 09, 2019

## Transcript

4. ### Mean-ﬁeld games (MFG) framework Introduced by P.-L. Lions, J.-M. Lasry

in the mathematics community and M. Huang, P. Caines, R. Malham´ e in the engineering one to model huge populations of identical, rational agents that play non-cooperative di↵erential games. In a PDE form, MFG resemble macroscopic equations from statistical physics in game-theoretic/optimal control framework. Yield near optimal solutions for huge populations of optimizing agents. Thus, tremendous simpliﬁcation in practical problems. engineer ﬁeld !4
5. ### Nash equilibrium in a differential game A continuum of players

with a distribution density ⇢(x, t) play a non-cooperative di↵erential game. An individual agent faces an optimization problem (x, t) = inf E t Z 0 L(Xs, vs, ⇢(·, s), s)ds + 0(X0, ⇢(·, 0)), (9) where dXs = vsds + p 2 dWs, XT = x, and L is the Lagrangian. In Nash equilibrium, each agent cannot do better unilaterally. Optimal strategies are given by the value function, , that solves the corresponding Hamilton-Jacobi equation. The density, ⇢(x, t), evolves according to optimal actions of agents. Thus, the Fokker-Planck equation. !5
6. ### The PDE formulation A typical time-dependent MFG system (derived from

(9)) has the form 8 > > > > > < > > > > > : t 2 + H (x, Dx (x, t), ⇢(x, t), t) = 0 ⇢t 2 ⇢ divx (⇢DpH (x, Dx (x, t), ⇢(x, t), t)) = 0 ⇢ 0, R Td ⇢(x, t)dx = 1, t 2 [0, T] ⇢(x, T) = ⇢T (x), (x, 0) = 0(x, ⇢(x, 0)), x 2 Td . (10) Given: H : Td ⇥ Rd ⇥ X ⇥ R ! R is a periodic in space and convex in momentum Hamiltonian, where X = R+ or X = L 1(Td ; R+) or X = R+ ⇥ L 1(Td ; R+). H = L ⇤ is a di↵usion parameter 0 : Td ⇥ X ! R, ⇢T : Td ! R given initial-terminal conditions Unknowns: , ⇢ : Td ⇥ [0, T] ! R above !6
7. ### Classiﬁcation of MFG Mean-ﬁeld interaction or non-linearity is the dependence

of H, L on the density, ⇢. Local interaction if H, L depend on ⇢ locally. For instance, H(x, p, ⇢, t) = ˜ H(x, p, t) ln ⇢ H(x, p, ⇢, t) = ˜ H(x, p, t) ⇢ Non-local interaction if H, L depend on ⇢ non-locally. For instance, H(x, p, ⇢, t) = ˜ H(x, p, t) F 0 @ x, Z Td K(x, y)⇢(y)dy 1 A Congestion interaction if we penalize moving in dense area. For instance (soft congestion) H(x, p, ⇢, t) = ⇢↵ ˜ H ✓ x, p ⇢↵ , ⇢, t ◆ , ↵ > 0 !7
8. ### U(t, µ) = inf ⇢2P(Td) F(⇢) + dW (⇢, µ)2

2t Hopf-Lax on density space E.g. (Burgers’) Hamilton-Jacobi on density space Characteristics on density space (Nash equilibrium in mean ﬁeld games) @tµs + r · (µs r s) = 0 @t s + 1 2 (r s)2 = 0 Math Review: mean ﬁeld games @ @t U(t, µ) + 1 2 Z Td (r µ U(t, µ))2µ(x)dx = 0, U(0, µ) = F(µ) !8
9. ### Density Control: Optimal Transport dW (⇢, µ)2 = inf ⇢s,us

n Z 1 0 Z R u2 s ⇢sdxdt: @s⇢s + r · (⇢sus) = 0, ⇢0 = ⇢, ⇢1 = µ o 8 What happens if the total population density is not normalized/balanced? 9

!11

14. ### 0 1 0 Density (7(x; t)) Time (t) 0.5 0.5

0.5 Space (x) 1 0 1 14
15. ### Unnormalized L1 Wasserstein metric Let p = 1: UW1(µ0, µ1)

= inf v,f(t) n Z 1 0 Z ⌦ kvkµdxdt + 1 ↵ Z 1 0 |f(t)|dt · |⌦|: @tµ + r · (µv) = f(t) o . 15 15
16. ### Time independent solution Denote m(x) = Z 1 0 v(t,

x)µ(t, x)dt, with the fact Z 1 0 f(t)dt = c = 1 |⌦| ⇣ Z ⌦ µ1(x)dx Z ⌦ µ0(x)dx ⌘ . then by Jensen’s inequality and integrating the time variable t, we obtain UW1(µ0, µ1) = inf m n Z ⌦ km(x)kdx + 1 ↵ Z ⌦ µ0(x)dx Z ⌦ µ1(x)dx : µ1(x) µ0(x) + r · m(x) = 1 |⌦| ⇣Z ⌦ µ1(x)dx Z ⌦ µ0(x)dx ⌘o . 16 From this equation, we see the momentum is independent of alpha. 16
17. ### Closed form solution In one space dimension on the interval

⌦ = [0, 1], the L1 unnormalized Wasserstein metric has the following explicit solution: UW1(µ0, µ1) = Z ⌦ Z x 0 µ1(y)dy Z x 0 µ0(y)dy x Z ⌦ (µ1(z) µ0(z))dz dx + 1 ↵ ⇣ Z ⌦ µ1(z)dz Z ⌦ µ0(z)dz ⌘ . 17 If the total masses are equal, the additional term disappears. 17
18. ### Unnormalized L2 Wasserstein metric Let p = 2: UW2(µ0, µ1)

= inf v,f(t) n Z 1 0 Z ⌦ kvk2µdxdt + 1 ↵ Z 1 0 f(t) 2dt · |⌦|: @tµ + r · (µv) = f(t) o . 22 18
19. ### Minimizer system The minimizer (v(t, x), µ(t, x), f(t)) for

UOT problem satisﬁes v(t, x) = r (t, x), f(t) = ↵ 1 |⌦| Z ⌦ (t, x)dx, and 8 > > > > < > > > > : @tµ(t, x) + r · (µ(t, x)r (t, x)) = ↵ 1 |⌦| Z ⌦ (t, x)dx @t (t, x) + 1 2 kr (t, x)k2 = 0 µ(0, x) = µ0(x), µ(1, x) = µ1(x). 23 19
20. ### Unnormalized Monge-Ampere equation Denote (x) = 1 2 kxk2 +

(0, x), Following the Hopf-Lax formula, the minimizer of unnormalized OT satisﬁes µ(1, r (x))Det(r2 (x)) µ(0, x) =↵ Z 1 0 Det ⇣ tr2 (x) + (1 t)I ⌘ · Z ⌦ ⇣ (y) kyk2 2 + tkr (y) yk2 2 ⌘ Det ⇣ tr2 (y) + (1 t)I ⌘ dydt. 24 20
21. ### Unnormalized Kantorovich problem 1 2 UW2(µ0, µ1) 2 = sup

n Z ⌦ (1, x)µ(1, x)dx Z ⌦ (0, x)µ(0, x)dx ↵ 2 Z 1 0 ⇣ Z ⌦ (t, x)dx ⌘2 dt o where the supremum is taken among all : [0, 1] ! ⌦ satisfying @t (t, x) + 1 2 kr (t, x)k2  0. 25 This is connected to one of UCLA’s fast algorithms for OT by M. Jacobs and F. Leger, 2019. 21

27. ### Generalized unnormalized OT In fact, when p=2, the minimization has

the unconstrained optimization formulation: Models suggested by Chen, Georgiou, Ling, Tannenbaum, 2017 for p=1,2. where Lµ = r · (µr) !27

32. ### Mean ﬁeld game on manifolds Here we replace the domain

of MFG from torus strategy set to a manifold M, point clouds, and graphs (discrete strategy set) U(t, µ) = inf v,µ,f Z 1 0 Z M kv(t, x)kpµ(t, x) F(µ(t, ·))dxdt + G(µ(0, ·)) such that the dynamical constraint is satisﬁed @tµ + r · (µv) = 0, µ(1, x) = µ1(x) Consider !32

34. ### MFG in 1D via unconstrained optimization Evolution of density with

Phi(0,x) ﬁxed !34
35. ### Classical MFG examples via unconstrained optimization The evolution of density

Quadratic interaction energy with linear periodic potential energy We test our algorithms using examples in Achdou, Capuzzo-Dolcetta, Mean Field Games: Numerical Methods, 2009. !35
36. ### Classical MFG examples via unconstrained optimization The evolution of density:

quadratic interaction energy with linear periodic potential energy !36
37. ### Spectral methods in nonlocal MFG Goal: Solve MFG with nonlocal

interactions 8 > < > : t + D + H(x, r ) = R ⌦ K(x, y)⇢(y, t)dy, ⇢t + D⇤⇢ div(⇢rpH(x, r )) = 0, (x, 0) = 0(x), ⇢(x, T) = ⇢T (x). (13) Above, K is a monotone symmetric interaction kernel, D is a monotone linear operator (e.g. D = ). Strategy: Model the interaction in a Fourier space. Speciﬁcally, given a basis {x 7! fi (x)}i expand the kernel K(x, y) = X i,j kij fi (x)fj (y). !37
38. ### Furthermore, Z ⌦ K(x, y)⇢(y, t)dy = X i ai

(t)fi (x), where ai (t) = X j kij Z ⌦ ⇢(y, t)fi (y)dy. Hence, (13) can be written as 8 > > > > < > > > > : t + D + H(x, r ) = P i ai (t)fi (x), ⇢t + D⇤⇢ div(⇢rpH(x, r )) = 0, ai (t) = P j kij R ⌦ ⇢(y, t)fi (y)dy, 8i, (x, 0) = 0(x), ⇢(x, T) = ⇢T (x). (14) The problem !38
39. ### New variational principle: System (14) is equivalent to inf a

⇢ a · K 1 a 2 Z ⌦ a(x, T)⇢T (x)dx s.t. (16) holds , (15) where K = (kij ), and a is the viscosity solution of ( t + D + H(x, r ) = P i ai (t)fi (x), (x, 0) = 0(x). (16) Advantages: (15) is a convex optimization problem! Flexible and sparse representations of mean-ﬁeld interactions Grid-free approximation methods for ﬁrst-order problems Compatibility with powerful convex optimization techniques such as ADMM, PDHG the system It is !39
40. ### Examples: Take a periodic domain ⌦ = T2, and periodic

Gaussian interaction kernels K ,µ (x, y) = 2µ2 ⇡ 2 X ↵2Z2 e 2|x y ↵|2 2 , x, y 2 T2. (17) Furthermore, suppose that H(x, p) = |p|2 2 , (x, p) 2 T2 ⇥ R2, D = 0. (18) Observation: Trigonometric system yields diagonal (sparse) representations for translation invariant kernels! !40
41. ### Figure: Gaussian kernels for ( , µ) = (0.1, 0.75),

(0.1, 0.5), (1, 0.5). Figure: The terminal distribution ⇢T (x1, x2 ) and the initial cost 0 (x1, x2 ). !41

48. ### Constrained MFG Given a parameterized density ⇢(✓, x) , consider

the variational problem @t⇢(✓(t), x) + r · (⇢(✓(t), x)v(t, x)) = 0, ✓(T) ﬁxed such that the constrained continuity equation holds Typical parameterized density are given by probability models, such as mixed Gaussian, deep neural networks, graphical models etc. inf ⇢(✓,·) Z T 0 n Z L(x, v)⇢(✓(t), x)dx F(⇢(✓(t), ·)) o dt + G(⇢(✓(0), ·)) !48

50. ### Evolutionary dynamics for Robotics min ⇢ KL(⇢k⇢ref ) + 1

2 Z Z W(x, y)⇢(x)⇢(y)dxdy Wasserstein Gradient ﬂow @t⇢ = r · (⇢rW ⇤ ⇢) + r · (⇢r log ⇢ ⇢ref ) Lagrangian coordinates ˙ X = rW ⇤ ⇢ + r log ⇢ ⇢ref !50