DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit Poker

DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit Poker

PWL SF Talk

7c4bac30ed2d3a9d346ced746b1d985d?s=128

Tom Santero

March 30, 2017
Tweet

Transcript

  1. DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit Poker @tsantero PWL

    SF March 30, 2017
  2. Texas Hold’em "truly a 'sawdust joint, with…oiled sawdust covering the

    floors." originated in Texas one of many variations
 of the game of poker most widely played card
 game in the world
  3. Texas Hold’em originated in Texas one of many variations
 of

    the game of poker most widely played card
 game in the world
  4. 1 2 3 4 5 6 7 8 9 10

  5. 1 2 3 4 5 6 7 8 9 10

    gameplay moves clockwise
  6. D SB BB Blinds: 1/2

  7. D SB BB

  8. BB D SB

  9. SB BB D

  10. flop turn river hole cards ∀players player actions: check, bet,

    raise, fold
  11. “The honor’s in the dollar, kid.”
 —Seth Davis, Boiler Room

    (2000) Keeping Score in Hold’em ________________ Cost of Big Blind Chip Stack = # BBs ________________ 100 BB = mBB
  12. For example…..

  13. tsantero’s hand cowboy’s hand Preflop: 
 I raise to $20


    cowboy raises to $45 I call all other players fold Odds of flopping:
 Trips 29 to 1 Straight 75 to 1 Flush 118 to 1 6 6 5 5 ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠
  14. tsantero’s hand 6 6 5 5 ♠ ♠ ♠ ♠

    ♠ ♠ ♠ ♠ ♠ ♠ ♠ 3 3 2 2 4 4 ♣ ♣ ♣ ❤ ❤ ♦ ♦ ♦ ♦ pot size: $90 I bet $65 cowboy calls
  15. tsantero’s hand 6 6 5 5 ♠ ♠ ♠ ♠

    ♠ ♠ ♠ ♠ ♠ ♠ ♠ 3 3 2 2 4 4 ♣ ♣ ♣ ❤ ❤ ♦ ♦ ♦ ♦ pot size: $210 I shove all in ($420) cowboy calls J J
  16. tsantero’s hand 6 6 5 5 ♠ ♠ ♠ ♠

    ♠ ♠ ♠ ♠ ♠ ♠ ♠ 3 3 2 2 4 4 ♣ ♣ ♣ ❤ ❤ ♦ ♦ ♦ ♦ J J cowboy’s hand 3 3 ♠ ♠ ♠ 3 3 ♦ ♦ ♦ 4/38 ~ 10%
  17. tsantero’s hand 6 6 5 5 ♠ ♠ ♠ ♠

    ♠ ♠ ♠ ♠ ♠ ♠ ♠ 3 3 2 2 4 4 ♣ ♣ ♣ ❤ ❤ ♦ ♦ ♦ ♦ J J cowboy’s hand 3 3 ♠ ♠ ♠ 3 3 ♦ ♦ ♦ J J
  18. None
  19. 
 Imperfect information games require more complex reasoning than similarly

    sized perfect information games. The correct decision at a particular moment depends upon the probability distribution over private information that the opponent holds, which is revealed through their past actions.

  20. Heads-Up No Limit Texas Holdem: total number of decision points:

    10160 (10170 in Go) past approaches include pre-solving the entire game or abstraction via Counterfactual Regret Minimization (CFR) DeepStack: attempts to solve for a Nash Equilibrium
  21. It’s time for some game theory.

  22. Nash Equilibrium Let (S, f) represent a game with n

    players Si = strategy for player i such that: S = S1 x S2 x S3 … x Sn
 is the set of strategy profiles f(x) = (f1(x), f2(x), …fn(x)) is the payoff function
 evaluated at x ∈ S
  23. Nash Equilibrium contd. Let xi equal S for player i

    and x-i equal S for all other players IFF player i ∈ {1, … n} chooses strategy xi then payout function = fi(x) ; it follows that x* ∈ S will produce a Nash Equilibrium such that ∀i, xi ∈ Si : fi(xi*, x-i*) ≥ fi(xi*, x-i*)
  24. Nash Equilibrium contd. ∀i, xi ∈ Si : fi(xi*, x-i*)

    ≥ fi(xi*, x-i*) the strategy profile will produce a Nash Equilibrium so long as no unilateral deviation from the strategy by any single player results in an improved payoff function
  25. CONFESS CONFESS LIE LIE -6, -6 0, -10 -10, 0

    -2, -2 Prisoner 2 Prisoner 1
  26. Oskar Morgenstern and John von Neumann

  27. Oskar Morgenstern and John von Neumann a mixed strategy Nash

    Equilibrium will exist in any zero-sum game involving a finite set of actions
  28. a mixed strategy Nash Equilibrium will exist in any zero-sum

    game involving a finite set of actions K K K K decision to bet/call, raise, or fold will vary depending on the situation 5 5 ♠ ♠ ♠ ♠ ♠ K K J J
  29. a mixed strategy Nash Equilibrium will exist in any zero-sum

    game involving a finite set of actions K K K K decision to bet/call, raise, or fold will vary depending on the situation 5 5 ♦ ♦ ♦ ♦ A A J J ♣ ♦
  30. None
  31. Exploitable vs Non-exploitable play a player’s strategy: probability distribution over

    range of public actions
 determine a player’s range: function of the card’s they’ll play for what price expected utility: estimation of the payout function
  32. Exploitable vs Non-exploitable play a Nash Equilibrium strategy: seeks to

    maximize expected utility against a
 best-response opponent strategy exploitability: difference in expected utility of a best-response strategy and the expected utility under a NE
  33. DeepStack: limited exploitability via
 heuristic search local strategy: computation over

    the current public state depth-limited lookahead: learned value function restricted set of lookahead actions: difference in expected utility of a best-response strategy and the expected utility under a NE
  34. Continual Resolving initial state: strategy is determined by our range

    of play next actions: discard previous strategy and determine action using CFR algorithm. input: our stored range and a vector of opponent counterfactual values update our range and opponent counter factual values
  35. Rules for Resolving our action: replace opponent counterfactual values with

    those computed in the re-solved strategy for our chosen action update our own range using the computed strategy and Bayes’ rule chance action: replace the opponent counterfactual values with those computed for this chance action from the last resolve update our own range by zeroing out impossible hands given the new public state
  36. Limited Depth Lookahead “intuition” neural net: learned counterfactual function that

    approximates the resulting values if the public state were to be solved with the current iteration’s ranges input: public state = players’ ranges, pot size, public cards output: vector of counterfactual values for holding any hand for both players
  37. Sparse Lookahead Trees reduction of actions: generate a space lookahead

    tree limited to only 4 actions: fold, call, 2 or 3 bet, all-in optimization for decision speed: allows DeepStack to play at conventional human speed decision points reduced form 10160 to 107 can be solved in < 5 seconds using a GeForce GTX 1080
  38. Training the NN

  39. Results

  40. Results

  41. Questions?