Armed Bandits, Machine Learning, and Fast Java: Practical Advice for Real-Time APIs

Real-time APIs supply a wealth of access for data-driven platforms, offering scale at unprecedented reach and efficiency but increasingly narrow constraints. With an endless source of data but scarce CPU time, how is the Java framework well suited to service actionable data in real time? Which exploration-exploitation strategies minimize opportunity cost? How can supervised learning be adapted to fit the problem domain by leveraging Hadoop and Mahout? What is the relationship between parametric and nonparametric models? This presentation covers these questions and more from an engineering perspective. No particular mathematical or machine learning background is required.

Presented Tuesday, June 11, 2013 at JavaOne Shanghai.

Transcript

1. Armed Bandits, Machine Learning and Fast Java: Practical Advice for Real- Time APIs Breandan Considine OneSpot, Inc.
2. Why Real-Time? • The world is full of hard problems • Types of real time applications  Hard (nuclear reactor control)  Firm (auction bidding)  Soft (train scheduling) • Real-time is a good thing • Real world applications • Performance over scalability
3. Benefits of Real-Time Processing • Forces us to narrow our priorities • Focus on constant, stable solutions rather than time- varying, exact solutions • Abundance of data but scarce processing power  Lifespan of actionable data extremely short  Tradeoff between optimality and throughput • Speed and parallelism will come over time • Upfront investment with long-term benefits
4. Real-time Interactive Tasks (RITs) • Online auctions: DSPs, SSPs • Multivariate testing • Inventory control, SCM • Scheduling, navigation, routing • Recommendation systems • High frequency trading • Fraud prevention
5. Common Thread • Agent offered a context and set of choices • Each choice has a unknown payoff distribution • Choose an option, measure the outcome • Goal: Maximize cumulative payoff  Many instances  Time sensitive  Nontrivial features
6. Challenges • Impractical to test every action in context  Computationally intractable to consider  Cost of full survey outweighs benefit • Exploration-Exploitation Tradeoff  Opportunity cost for suboptimal choices  Local extrema conceal optimal solutions • Latency comes at the cost of throughput  Every clock cycle must count  Firm real-time characteristics
7. Traditional Supervised Learning Cycle
8. Reinforcement Learning (RL)
9. Dis/advantages • Starts from scratch, training is expensive • Credit assignment problem & reward structure • Issues with non-stationary systems • Continuously integrates feedback • Adapts to real-time decisions • No assumptions about data • Follows signal on-line • Similar to how we learn
10. Non-blocking Algorithms • Critical for high performance I/O • Relatively difficult to implement correctly • Offers large speedup over lock-based variants • Types of non-blocking guarantees  Wait-freedom  Lock-freedom  Obstruction-freedom
11. Lock-Freedom • Guarantees progress for at least one thread • Does not guarantee starvation-freedom • May be slower overall, see Amdahl's law
12. Java Memory Model • happens-before relation  Threaded operations follow a partial order  Ensures JVM does not reorder ops arbitrarily • Sequential consistency is guaranteed for race-free programs • Does not prevent threads from having different visibility on operations, unless explicitly declared
13. The volatile keyword • Mechanics governed by two simple rules  Each action within a thread happens in program order  volatile writes happen before all subsequent reads on that same field • Reads from and writes to main memory • Syntactic shorthand for lock on read, unlock on write – incurs similar performance toll
14. Java Concurrency • ConcurrentHashMap, ConcurrentLinkedQueue • Need to carefully benchmark  Can be significantly slower depending on implementation  Avoid using default hash map constructor  Faster implementations exist, lock-free  Java 8 improvements in the pipeline • Prone to atomicity violations
15. ConcurrentHashMap<String, Data> map; Data updateAndGet(String key) { Data d = map.get(key); if(d == null) { // Atomic violation d = new Data(); map.put(key, d); } return d; }
16. Java Atomics • Guarantees lock-free thread safety • Uses CAS primitives to ensure atomic execution • Better performance than volatile under low to moderate contention, must be tested in production setting
17. private T current; public synchronized <T> T compareAndSet(T expected, T new) { T previous = current; if(current == expected) current = new; return previous; }
18. ABA Problem • Direct equality testing is not sufficient • Full A-B-A transaction can execute immediately before execution of CAS primitive, causing unintended equality when structure has changed • Solution: generate a unique tag whenever value changes, then CAS against value-tag pair
19. False Sharing • Can be prevented by padding out fields • Java 8 addresses this problem with @Contended
20. Multi-Armed Bandit Problems • N choices, each with hidden payoff distributions • What strategy maximizes cumulative payoff? • Observation: Choose randomly from a distribution representing observed probability, return ARGMAX
21. Bayesian Bandits *http://camdp.com/blogs/multi-armed-bandits
22. Adaptive Control Problems • Parameter estimation for real time processes • Uses continuous feedback to adjust output
23. Pacing Techniques
24. PID Controller
25. Counting/Filtering Problems • Large domain of inputs (IPs, emails, strings) • Need to maintain online, streaming aggregates • See Hadoop libraries for good implementations • Observation: Fast hashing is key.
26. Bloom Filters • Fast probabilistic membership testing • Guarantees no false negatives, low space overhead
27. Special thanks to Ian Clarke Matt Cohen
28. References http://mechanical-sympathy.blogspot.ie/ http://camdp.com/blogs/multi-armed-bandits http://blog.locut.us/2011/09/22/proportionate-ab-testing http://blog.locut.us/2008/01/12/a-decent-stand-alone-java-bloom-filter-im plementation/ http://www.cl.cam.ac.uk/research/srg/netos/lock-free/ https://github.com/edwardw/high-scale-java-lib M. Michael, et al. Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms. [PDF] P. Tsigas, et al. Wait-free queue algorithms for the real-time java specification. [PDF]