Wykorzystanie klastraApache Mesos wdeploymencie aplikacjipythonowychKamil Warguła1github.com/quamilek[email protected]
View Slide
2
Apache Mesos3
What is Mesos?Apache Mesos abstracts CPU, memory,storage, and other compute resources awayfrom machines (physical or virtual),enabling fault-tolerant and elasticdistributed systems to easily be built and runeffectively.4
Mesos architectureMesosmasterStandbymasterStandbymasterZooKeeperquorumMesos slaveFrameworkexecutortaskMesos slaveFrameworkexecutortaskMesos slaveMarathonexecutortaskHadoopexecutortaskFrameworkscheduler5
Resource allocation6
Mesos UI7
Mesos frameworks8
Marathon9
MarathonMarathon is an Apache Mesosframework for long-runningapplications10
Marathon features● Web UI● JSON/REST API● HA● Constraints● Health Checks● Event Subscription● Basic Auth and SSL11
Resource utilization12
Resource utilization13
Mesos saves $$$14old way mesos way
simple python REST appimport falconimport jsonclass ExampleResource:def on_get(self, req, resp):data = {'foo': 'bar'}resp.body = json.dumps(data)api = falcon.API()api.add_route('/', ExampleResource())15
integration with gunicornfrom gunicorn.app.base import BaseApplicationclass StandaloneApplication(BaseApplication):...def _get_config():...def main():api = falcon.API()api.add_route('/', Example Resource())config = _get_config()StandaloneApplication(api, config).run()if __name__ == '__main__':main()16
How to runvirtualenv envpip install falcon==0.3.0 gunicorn==19.3.0source env/bin/activatepython api.py -p 809917
Pack requirements with PEXpex -v -r falcon==0.3.0 -r gunicorn==19.3.0 -o app.pex18
How to run app with pex./app.pex api.py -p 809919
DEMO20
Marathon UI21
Marathon app definition{"id": "appname","cmd": "python my_app.py","cpus": 0.5,"instances": 1,"mem": 64,"uris": ["http://artifacts.local/my_app_0.1.0.tar.gz"]}22
Marathon - deploy python appDEMO23
app configuration● environment variables● distributed configuration service(ZooKeeper, ETCD)24
Service Discovery● Marathon-Consul● Mesos-DNS25
Load Balancing● haproxy-marathon-bridge26
Scale app - manualimport jsonimport requestsurl = 'http://marathon.local/v2/apps/myapp'data = {'instances': 5}requests.put(url, data=json.dumps(data))27
Scale app - manualDEMO28
Autoscale appMIN_INSTANCES_COUNT = 2MAX_INSTANCES_COUNT = 20if overloaded_instances >= 0.9 * total_instances:total_instances += 2if total_instances <= MAX_INSTANCES_COUNT:scale_app(total_instances)elif overloaded_instances <= 0.5 * total_instances:total_instances -= 2if total_instances >= MIN_INSTANCES_COUNT:scale_app(total_instances)29
Autoscale appDEMO30
Thank you!31goo.gl/qWivrx
Q/A?32goo.gl/qWivrx