Making Distributed Computing Easy (Ion Stoica, Anyscale)

Ion Stoica, co-founder, executive chairman & president, Anyscale, highlights the product developments on the Anyscale platform, spotlights a few customer stories, and provides a sneak preview into the future.

Transcript

1. Making Distributed Computing Easy Ion Stoica Co-Founder, Executive Chairman & President, Anyscale Professor, UC Berkeley
2. Distributed apps are becoming the norm This Talk 01 Building distributed apps is very hard 02 Ray and Anyscale make developing, deploying and managing distributed apps easy 03
3. 01 02 Apps increasingly incorporate AI AI workloads are becoming distributed Distributed apps becoming the norm
4. 01 02 Apps increasingly incorporate AI AI workloads are becoming distributed Distributed apps becoming the norm 01
5. Apps increasingly incorporate AI
6. 01 02 Apps increasingly incorporate AI AI workloads are becoming distributed Distributed apps becoming the norm 01 02
7. https://openai.com/blog/ai-and-compute/ 35x every 18 m onths Compute demands, 2012-2019 (AI)
8. https://devblogs.nvidia.com/training-bert-with-gpus/ 2020 GPT-3 Compute demands, 2012-2020 (AI) No sign of slow down... 35x every 18 m onths
9. 35x every 18 m onths 2020 GPT-3 Growing gap between demand and supply Moore’s Law (2x every 18 months) CPU https://openai.com/blog/ai-and-compute/
10. 35x every 18 m onths 2020 GPT-3 Specialized hardware is not enough Moore’s Law (2x every 18 months) CPU https://openai.com/blog/ai-and-compute/ GPU* TPU * No way out but to distribute these apps!
11. 01 02 Apps becoming more and more complex Development ⇒ production is challenging Building distributed apps very hard!
12. 01 02 Apps becoming more and more complex Development ⇒ production is challenging Building distributed apps very hard! 01
13. Improve inference accuracy in fast changing environments Examples: recommendations, financial predictions, resource allocations Data Ingestion & Featurization Training Serving Online learning
14. Solution: stitch together a bunch of distributed systems Data Ingestion & Featurization Training Serving Online learning Serve
15. Data Ingestion & Featurization Training Serving Online learning Serve Hyper tuning
16. Training Serving Serve Simulations Environment Agent State/ reward Action Examples: industry automation, self-driving, trading & finance, system optimizations, recommendations, etc. Solution: stitch together a bunch of distributed systems (e.g., Facebook’s Horizon) Reinforcement learning
17. Training Serving Simulations Environment Agent Solution: build it from scratch (e.g. DeepMind’s Acme) Reinforcement learning State/ reward Action
18. Backend Business Logic Serving / Inference request reply Serving Solution: stitch together a bunch of distributed systems Backend: Business Logic & Inference
19. High performance, but very expensive • Time • People Few companies can afford, e.g., Google, Facebook, ... Challenges with building from scratch
20. Hard to develop: different APIs Hard to deploy & manage: impedance mismatch Slow: high overhead of moving data between different systems Data Processing Training Serving Hyper. Tuning Business Logic Simulations Serving KFServing Challenges with stitching together
21. Ray unifies distributed workloads
22. Data Processing Training Serving Hyper. Tuning Others Ray ecosystem + Native universal framework for distributed computing Business Logic
23. Data Processing Training Serving Hyper. Tuning Others Ray ecosystem + Native Best ecosystem of distributed libraries Instead of stitching systems, call libraries in same system Easy to develop, manage, and deploy Business Logic
24. Online learning Reinforcement learning Business Logic & Inference Framework for online learning Real examples at Ray Summit
25. 01 02 Apps becoming more and more complex Development ⇒ production is challenging Building distributed apps very hard! 01 02
26. Edit Run Debug Test / staging Deploy Development Production Application Lifecycle
27. Edit Run Debug Test / staging Deploy Development Production What do developers want ? Develop on your laptop Test and deploy on the cluster/cloud Laptop Cluster/cloud
28. Edit Run Debug Test / staging Deploy Development Production Develop on your laptop? Huge barrier - Local development and tests cannot reproduce cluster deployment - Need to package/dockerize app
29. Edit Run Debug Test / staging Deploy Development Production Develop on the cluster? Hard to develop on the cluster/cloud: no good tools, slow to launch nodes, expensive.
30. Anyscale simplifies development, deployment and management of Ray apps
31. universal framework for distributed computing Data Processing Training Serving Hyper. Tuning Business Logic & Simulations Others Ray ecosystem + Native
32. Edit Run Debug Test / staging Deploy Development Production Anyscale: Best of both worlds Laptop development experience and cloud scale
33. Edit Run Debug Development Development: Infinite laptop Can start developing on your laptop...
34. Edit Run Debug Development Development: Infinite laptop ... then transparently move to the cloud Like your laptop but with “infinite” resources!
35. Development Infinite laptop: How? NEW 1. Sync up local environment and files to the cloud Edit Run Debug
36. Infinite laptop: How? import ray ray.client().connect() ... >python ray_prog.py >RAY_ADDRESS=”anyscale://“ python ray_prog.py ray_prog.py run on laptop run in the cloud 1. Sync up local environment and files to the cloud 2. Run program in the cloud with no code changes NEW NEW
37. 1. Sync up local environment and files to the cloud 2. Run program in the cloud with no code changes 3. Serverless experience Development Infinite laptop: How? NEW NEW Edit Run Debug
38. 1. Sync up local environment and files to the cloud 2. Run program in the cloud with no code changes 3. Serverless experience 4. Debug programs like on your laptop Development Infinite laptop: How? NEW NEW NEW Edit Run Debug
39. Development → Production Edit Run Debug Test / staging Deploy Development Production
40. Development → Production Edit Run Debug Test / staging Deploy Development Production 1. App packaging NEW
41. Development → Production Edit Run Debug Test / staging Deploy Development Production 1. App packaging 2. SDK & REST APIs NEW NEW
42. Development → Production Edit Run Debug Test / staging Deploy Development Production 1. App packaging 2. SDK & REST APIs 3. Monitoring & observability NEW NEW
43. Demo