Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych

Transcript

1. Wykorzystanie klastra
   Apache Mesos w
   deploymencie aplikacji
   pythonowych
   Kamil Warguła
   1
   github.com/quamilek
   kamil@wargula.pl
   

   View Slide

2. View Slide

3. Apache Mesos
   3
   

   View Slide

4. What is Mesos?
   Apache Mesos abstracts CPU, memory,
   storage, and other compute resources away
   from machines (physical or virtual),
   enabling fault-tolerant and elastic
   distributed systems to easily be built and run
   effectively.
   4
   

   View Slide

5. Mesos architecture
   Mesos
   master
   Standby
   master
   Standby
   master
   ZooKeeper
   quorum
   Mesos slave
   Framework
   executor
   task
   Mesos slave
   Framework
   executor
   task
   Mesos slave
   Marathon
   executor
   task
   Hadoop
   executor
   task
   Framework
   scheduler
   5
   

   View Slide

6. Resource allocation
   6
   

   View Slide

7. Mesos UI
   7
   

   View Slide

8. Mesos frameworks
   8
   

   View Slide

9. Marathon
   9
   

   View Slide

10. Marathon
    Marathon is an Apache Mesos
    framework for long-running
    applications
    10
    

    View Slide

11. Marathon features
    ● Web UI
    ● JSON/REST API
    ● HA
    ● Constraints
    ● Health Checks
    ● Event Subscription
    ● Basic Auth and SSL
    11
    

    View Slide

12. Resource utilization
    12
    

    View Slide

13. Resource utilization
    13
    

    View Slide

14. Mesos saves $$$
    14
    old way mesos way
    

    View Slide

15. simple python REST app
    import falcon
    import json
    class ExampleResource:
    def on_get(self, req, resp):
    data = {'foo': 'bar'}
    resp.body = json.dumps(data)
    api = falcon.API()
    api.add_route('/', ExampleResource())
    15
    

    View Slide

16. integration with gunicorn
    from gunicorn.app.base import BaseApplication
    class StandaloneApplication(BaseApplication):
    ...
    def _get_config():
    ...
    def main():
    api = falcon.API()
    api.add_route('/', Example Resource())
    config = _get_config()
    StandaloneApplication(api, config).run()
    if __name__ == '__main__':
    main()
    16
    

    View Slide

17. How to run
    virtualenv env
    pip install falcon==0.3.0 gunicorn==19.3.0
    source env/bin/activate
    python api.py -p 8099
    17
    

    View Slide

18. Pack requirements with PEX
    pex -v -r falcon==0.3.0 -r gunicorn==19.3.0 -o app.pex
    18
    

    View Slide

19. How to run app with pex
    ./app.pex api.py -p 8099
    19
    

    View Slide

20. DEMO
    20
    

    View Slide

21. Marathon UI
    21
    

    View Slide

22. Marathon app definition
    {
    "id": "appname",
    "cmd": "python my_app.py",
    "cpus": 0.5,
    "instances": 1,
    "mem": 64,
    "uris": [
    "http://artifacts.local/my_app_0.1.0.tar.gz"
    ]
    }
    22
    

    View Slide

23. Marathon - deploy python app
    DEMO
    23
    

    View Slide

24. app configuration
    ● environment variables
    ● distributed configuration service
    (ZooKeeper, ETCD)
    24
    

    View Slide

25. Service Discovery
    ● Marathon-Consul
    ● Mesos-DNS
    25
    

    View Slide

26. Load Balancing
    ● haproxy-marathon-bridge
    26
    

    View Slide

27. Scale app - manual
    import json
    import requests
    url = 'http://marathon.local/v2/apps/myapp'
    data = {'instances': 5}
    requests.put(url, data=json.dumps(data))
    27
    

    View Slide

28. Scale app - manual
    DEMO
    28
    

    View Slide

29. Autoscale app
    MIN_INSTANCES_COUNT = 2
    MAX_INSTANCES_COUNT = 20
    if overloaded_instances >= 0.9 * total_instances:
    total_instances += 2
    if total_instances <= MAX_INSTANCES_COUNT:
    scale_app(total_instances)
    elif overloaded_instances <= 0.5 * total_instances:
    total_instances -= 2
    if total_instances >= MIN_INSTANCES_COUNT:
    scale_app(total_instances)
    29
    

    View Slide

30. Autoscale app
    DEMO
    30
    

    View Slide

31. Thank you!
    31
    goo.gl/qWivrx
    

    View Slide

32. Q/A?
    32
    goo.gl/qWivrx
    

    View Slide