Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych

Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych

Kamil Warguła

November 29, 2015
Tweet

Other Decks in Technology

Transcript

  1. Wykorzystanie klastra
    Apache Mesos w
    deploymencie aplikacji
    pythonowych
    Kamil Warguła
    1
    github.com/quamilek
    [email protected]

    View Slide

  2. 2

    View Slide

  3. Apache Mesos
    3

    View Slide

  4. What is Mesos?
    Apache Mesos abstracts CPU, memory,
    storage, and other compute resources away
    from machines (physical or virtual),
    enabling fault-tolerant and elastic
    distributed systems to easily be built and run
    effectively.
    4

    View Slide

  5. Mesos architecture
    Mesos
    master
    Standby
    master
    Standby
    master
    ZooKeeper
    quorum
    Mesos slave
    Framework
    executor
    task
    Mesos slave
    Framework
    executor
    task
    Mesos slave
    Marathon
    executor
    task
    Hadoop
    executor
    task
    Framework
    scheduler
    5

    View Slide

  6. Resource allocation
    6

    View Slide

  7. Mesos UI
    7

    View Slide

  8. Mesos frameworks
    8

    View Slide

  9. Marathon
    9

    View Slide

  10. Marathon
    Marathon is an Apache Mesos
    framework for long-running
    applications
    10

    View Slide

  11. Marathon features
    ● Web UI
    ● JSON/REST API
    ● HA
    ● Constraints
    ● Health Checks
    ● Event Subscription
    ● Basic Auth and SSL
    11

    View Slide

  12. Resource utilization
    12

    View Slide

  13. Resource utilization
    13

    View Slide

  14. Mesos saves $$$
    14
    old way mesos way

    View Slide

  15. simple python REST app
    import falcon
    import json
    class ExampleResource:
    def on_get(self, req, resp):
    data = {'foo': 'bar'}
    resp.body = json.dumps(data)
    api = falcon.API()
    api.add_route('/', ExampleResource())
    15

    View Slide

  16. integration with gunicorn
    from gunicorn.app.base import BaseApplication
    class StandaloneApplication(BaseApplication):
    ...
    def _get_config():
    ...
    def main():
    api = falcon.API()
    api.add_route('/', Example Resource())
    config = _get_config()
    StandaloneApplication(api, config).run()
    if __name__ == '__main__':
    main()
    16

    View Slide

  17. How to run
    virtualenv env
    pip install falcon==0.3.0 gunicorn==19.3.0
    source env/bin/activate
    python api.py -p 8099
    17

    View Slide

  18. Pack requirements with PEX
    pex -v -r falcon==0.3.0 -r gunicorn==19.3.0 -o app.pex
    18

    View Slide

  19. How to run app with pex
    ./app.pex api.py -p 8099
    19

    View Slide

  20. DEMO
    20

    View Slide

  21. Marathon UI
    21

    View Slide

  22. Marathon app definition
    {
    "id": "appname",
    "cmd": "python my_app.py",
    "cpus": 0.5,
    "instances": 1,
    "mem": 64,
    "uris": [
    "http://artifacts.local/my_app_0.1.0.tar.gz"
    ]
    }
    22

    View Slide

  23. Marathon - deploy python app
    DEMO
    23

    View Slide

  24. app configuration
    ● environment variables
    ● distributed configuration service
    (ZooKeeper, ETCD)
    24

    View Slide

  25. Service Discovery
    ● Marathon-Consul
    ● Mesos-DNS
    25

    View Slide

  26. Load Balancing
    ● haproxy-marathon-bridge
    26

    View Slide

  27. Scale app - manual
    import json
    import requests
    url = 'http://marathon.local/v2/apps/myapp'
    data = {'instances': 5}
    requests.put(url, data=json.dumps(data))
    27

    View Slide

  28. Scale app - manual
    DEMO
    28

    View Slide

  29. Autoscale app
    MIN_INSTANCES_COUNT = 2
    MAX_INSTANCES_COUNT = 20
    if overloaded_instances >= 0.9 * total_instances:
    total_instances += 2
    if total_instances <= MAX_INSTANCES_COUNT:
    scale_app(total_instances)
    elif overloaded_instances <= 0.5 * total_instances:
    total_instances -= 2
    if total_instances >= MIN_INSTANCES_COUNT:
    scale_app(total_instances)
    29

    View Slide

  30. Autoscale app
    DEMO
    30

    View Slide

  31. Thank you!
    31
    goo.gl/qWivrx

    View Slide

  32. Q/A?
    32
    goo.gl/qWivrx

    View Slide