Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych

Transcript

1. Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych Kamil Warguła 1 github.com/quamilek kamil@wargula.pl
3. Apache Mesos 3
4. What is Mesos? Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. 4
5. Mesos architecture Mesos master Standby master Standby master ZooKeeper quorum Mesos slave Framework executor task Mesos slave Framework executor task Mesos slave Marathon executor task Hadoop executor task Framework scheduler 5
6. Resource allocation 6
7. Mesos UI 7
8. Mesos frameworks 8
9. Marathon 9
10. Marathon Marathon is an Apache Mesos framework for long-running applications 10
11. Marathon features • Web UI • JSON/REST API • HA • Constraints • Health Checks • Event Subscription • Basic Auth and SSL 11
12. Resource utilization 12
13. Resource utilization 13
14. Mesos saves $$$ 14 old way mesos way
15. simple python REST app import falcon import json class ExampleResource: def on_get(self, req, resp): data = {'foo': 'bar'} resp.body = json.dumps(data) api = falcon.API() api.add_route('/', ExampleResource()) 15
16. integration with gunicorn from gunicorn.app.base import BaseApplication class StandaloneApplication(BaseApplication): ... def _get_config(): ... def main(): api = falcon.API() api.add_route('/', Example Resource()) config = _get_config() StandaloneApplication(api, config).run() if __name__ == '__main__': main() 16
17. How to run virtualenv env pip install falcon==0.3.0 gunicorn==19.3.0 source env/bin/activate python api.py -p 8099 17
18. Pack requirements with PEX pex -v -r falcon==0.3.0 -r gunicorn==19.3.0 -o app.pex 18
19. How to run app with pex ./app.pex api.py -p 8099 19
20. DEMO 20
21. Marathon UI 21
22. Marathon app definition { "id": "appname", "cmd": "python my_app.py", "cpus": 0.5, "instances": 1, "mem": 64, "uris": [ "http://artifacts.local/my_app_0.1.0.tar.gz" ] } 22
23. Marathon - deploy python app DEMO 23
24. app configuration • environment variables • distributed configuration service (ZooKeeper, ETCD) 24
25. Service Discovery • Marathon-Consul • Mesos-DNS 25
26. Load Balancing • haproxy-marathon-bridge 26
27. Scale app - manual import json import requests url = 'http://marathon.local/v2/apps/myapp' data = {'instances': 5} requests.put(url, data=json.dumps(data)) 27
28. Scale app - manual DEMO 28
29. Autoscale app MIN_INSTANCES_COUNT = 2 MAX_INSTANCES_COUNT = 20 if overloaded_instances >= 0.9 * total_instances: total_instances += 2 if total_instances <= MAX_INSTANCES_COUNT: scale_app(total_instances) elif overloaded_instances <= 0.5 * total_instances: total_instances -= 2 if total_instances >= MIN_INSTANCES_COUNT: scale_app(total_instances) 29
30. Autoscale app DEMO 30
31. Thank you! 31 goo.gl/qWivrx
32. Q/A? 32 goo.gl/qWivrx