Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych
Search
Kamil Warguła
November 29, 2015
Technology
0
280
Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych
Kamil Warguła
November 29, 2015
Tweet
Share
Other Decks in Technology
See All in Technology
日本におけるデータエンジニアリングのこれまでとこれから
foursue
16
4.2k
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
160
非同期推論システムによるコスト削減と信頼性向上
koki_nishihara
0
160
TechFeed Experts Night#27 〜 フロントエンドフレームワーク最前線 (Svelte)
baseballyama
1
190
開発生産性大幅アップ!Postman VS Code拡張機能
nagix
2
360
オーナーシップを持つ領域を明確にする
konifar
13
3.1k
サーバー間 GraphQL と webmock-graphql の話 / server-to-server graphql and webmock-graphql
qsona
2
180
本当のAWS基礎
toru_kubota
0
490
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
5
410
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
2
440
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.3k
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
Featured
See All Featured
Automating Front-end Workflow
addyosmani
1356
200k
Building an army of robots
kneath
300
41k
Designing for humans not robots
tammielis
248
25k
The Illustrated Children's Guide to Kubernetes
chrisshort
31
46k
Building a Scalable Design System with Sketch
lauravandoore
456
32k
10 Git Anti Patterns You Should be Aware of
lemiorhan
648
58k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
116
18k
How GitHub (no longer) Works
holman
304
140k
The Cost Of JavaScript in 2023
addyosmani
16
3.8k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
What the flash - Photography Introduction
edds
64
11k
Transcript
Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych Kamil Warguła
1 github.com/quamilek
[email protected]
2
Apache Mesos 3
What is Mesos? Apache Mesos abstracts CPU, memory, storage, and
other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. 4
Mesos architecture Mesos master Standby master Standby master ZooKeeper quorum
Mesos slave Framework executor task Mesos slave Framework executor task Mesos slave Marathon executor task Hadoop executor task Framework scheduler 5
Resource allocation 6
Mesos UI 7
Mesos frameworks 8
Marathon 9
Marathon Marathon is an Apache Mesos framework for long-running applications
10
Marathon features • Web UI • JSON/REST API • HA
• Constraints • Health Checks • Event Subscription • Basic Auth and SSL 11
Resource utilization 12
Resource utilization 13
Mesos saves $$$ 14 old way mesos way
simple python REST app import falcon import json class ExampleResource:
def on_get(self, req, resp): data = {'foo': 'bar'} resp.body = json.dumps(data) api = falcon.API() api.add_route('/', ExampleResource()) 15
integration with gunicorn from gunicorn.app.base import BaseApplication class StandaloneApplication(BaseApplication): ...
def _get_config(): ... def main(): api = falcon.API() api.add_route('/', Example Resource()) config = _get_config() StandaloneApplication(api, config).run() if __name__ == '__main__': main() 16
How to run virtualenv env pip install falcon==0.3.0 gunicorn==19.3.0 source
env/bin/activate python api.py -p 8099 17
Pack requirements with PEX pex -v -r falcon==0.3.0 -r gunicorn==19.3.0
-o app.pex 18
How to run app with pex ./app.pex api.py -p 8099
19
DEMO 20
Marathon UI 21
Marathon app definition { "id": "appname", "cmd": "python my_app.py", "cpus":
0.5, "instances": 1, "mem": 64, "uris": [ "http://artifacts.local/my_app_0.1.0.tar.gz" ] } 22
Marathon - deploy python app DEMO 23
app configuration • environment variables • distributed configuration service (ZooKeeper,
ETCD) 24
Service Discovery • Marathon-Consul • Mesos-DNS 25
Load Balancing • haproxy-marathon-bridge 26
Scale app - manual import json import requests url =
'http://marathon.local/v2/apps/myapp' data = {'instances': 5} requests.put(url, data=json.dumps(data)) 27
Scale app - manual DEMO 28
Autoscale app MIN_INSTANCES_COUNT = 2 MAX_INSTANCES_COUNT = 20 if overloaded_instances
>= 0.9 * total_instances: total_instances += 2 if total_instances <= MAX_INSTANCES_COUNT: scale_app(total_instances) elif overloaded_instances <= 0.5 * total_instances: total_instances -= 2 if total_instances >= MIN_INSTANCES_COUNT: scale_app(total_instances) 29
Autoscale app DEMO 30
Thank you! 31 goo.gl/qWivrx
Q/A? 32 goo.gl/qWivrx