[Dalhousie University, 2016] Symbiotic Bid-based (SBB) GP

Transcript

1. Practical Experiences with SBB for Reinforcement Learning Jéssica Pauli de C. Bonson, NIMS Lab https://goo.gl/Whtp9n
2. Index 1. Quick SBB Overview 2. Case Studies 3. Practical Application of SBB to the Cases
3. SBB: General Idea • teams of programs evolve and compete over time • each team represents a player in the game • at each generation they play many matches against various opponents • the best ones reproduce and keep evolving, the others are discarded
4. Genetic Algorithm
5. SBB framework
6. • each team is composed of 2 or more programs • programs are composed of a set of instructions, registers and an action • before a team executes an action: ◦ all of its programs run over the inputs for the current match state ◦ the action from the program with the highest output is selected as the team’s action How the teams work?
7. Case Studies • TicTacToe • Poker
8. Example: TicTacToe • inputs: 9 • actions: 9 • opponents: Random and Smart • points: tictactoe matches’ states + opponents
9. Example: Poker • inputs: 14 (hand strength, position, pot odds, opponent model...) • actions: 3 • opponents: ◦ dummy: AlwaysRaise, AlwaysCall, AlwaysFold, Random ◦ static: LooseAggressive, LoosePassive, TightAggressive, TightPassive ◦ dynamic: Bayesian • points: poker matches’ states + opponents
10. Code • SBB for reinforcement learning: ◦ https://github.com/jpbonson/SBBReinforcementLe arner • SBB for classification: ◦ https://github.com/jpbonson/SBBClassifier • Warning! The code is under development!
11. Practical Application of SBB to the Cases Studies
12. Fitness Function • For RL, usually it is: ◦ Win: 1.0 ◦ Draw: 0.5 ◦ Lose: 0.0 • But depending on the game, other functions are able to give more information ◦ eg.: chips won/lose (poker)
13. Inputs • Very important: They define what the teams can see about the environment. • Can be used to control for a weak or strong AI. • Tweak: SBB deals better with inputs normalized between 0.0-10.0 instead of 0.0-1.0.
14. Opponents • Too easy opponents: ◦ Teams learn to beat them quickly and stop evolving • Too strong opponents: ◦ Teams aren’t able to learn to beat them, and SBB turns into a random walk • It is important to balance • Hall of Fame can be used to avoid evolutionary forgetting, but don’t overuse.
15. Points • How many? ◦ The more, the better ◦ But it is time consuming • It is important to invest time on optimizing the matches • Ensure all teams see the exact same points • For some games the points can’t be just like the real world task
16. Diversity • Important to avoid overfitting, so teams can continue to evolve new behaviors • Two methods: ◦ Point Profiles ◦ Pareto Dominance + Diversity metrics
17. Diversity: Point Profiles • Profile: the results of a team run against a set of sample inputs, representing states of the game • The new teams are mutated until their profile is different from their parent and/or all the other teams • The set of sample inputs can be made manually or during the training
18. Diversity: Pareto + Diversity • Pareto Dominance: ◦ A method to choose teams so that the chosen ones are the best regarding both the fitness function and the diversity metric
19. Source: http://pt.slideshare.net/abedsayyad/sayyad-slides-raise13
20. Diversity: Diversity Metrics • There are a lot of diversity metrics available, both general metrics and domain-specific metrics • Two types: ◦ Genotype ◦ Phenotype ▪ distance measures (euclidean, hamming...) ▪ entropy ▪ NCD
21. Second Layer • Uses the teams trained in the first layer as actions for the teams in the second layer • Goal: More complex behavior using specialized actions ◦ Eg.: instead of call/raise/fold for poker, it could be passive/aggressive behavior
22. Mutation Rates • Seven types of mutation rate ◦ Remove program from team, swap instructions in programs, etc… • Both passive and aggressive mutations work • It is an exploitation/exploration trade-off
23. Registers • Varies according to the game • For TicTacToe, using 5 instead of 2 improved the score by around 20%. • It is important to reset the registers between matches, but during a match they can be used as memory
24. Team Size + Program Size • Trade-off between complexity and runtime • Varies according to the game ◦ Team size: 2-9 for TTT, 2-16 for poker, +-30 for soccer ◦ Program size: 2-20 for TTT, 5-40 for poker • Should be big enough to deal with the complexity of the game and have space for introns
25. Operators • Simple set: +, -, /, * • Complex set: ln, exp, cos, sin • Ifs set: >=, < • Trade-off between complexity and runtime • Solution for overflow: Rollback so the target register isn’t modified by the instruction
26. Be able to Reuse Teams • It is important so you are able to: ◦ run them against various test cases after training ◦ integrate a trained team as an AI in a system ◦ use them as actions in a second layer • Solution: Save teams as .json files
27. Sample .json of trained poker team
28. Sample of a program with and without introns
29. Metrics • Most of the time runs will take a long time to finish, so it is important to think about what metrics are necessary and code them beforehand • Automated metrics save a lot of time • Be careful with bugs
30. Tests + Inheritance • pSBB for Reinforcement Learning has around 300k lines of code and around 40 classes • Automated tests and inheritance are essential.
31. Questions?