Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Wykorzystanie klastra Apache Mesos w deploymenc...
Search
Kamil Warguła
November 29, 2015
Technology
0
330
Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych
Kamil Warguła
November 29, 2015
Tweet
Share
Other Decks in Technology
See All in Technology
LLM時代にデータエンジニアの役割はどう変わるか?
ikkimiyazaki
4
780
社内報はAIにやらせよう / Let AI handle the company newsletter
saka2jp
5
610
定期的な価値提供だけじゃない、スクラムが導くチームの共創化 / 20251004 Naoki Takahashi
shift_evolve
PRO
3
330
KMP の Swift export
kokihirokawa
0
340
生成AIを活用したZennの取り組み事例
ryosukeigarashi
0
210
Optuna DashboardにおけるPLaMo2連携機能の紹介 / PFN LLM セミナー
pfn
PRO
1
890
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
Trust as Infrastructure
bcantrill
0
350
GopherCon Tour 概略
logica0419
2
190
「Verify with Wallet API」を アプリに導入するために
hinakko
1
240
スタートアップにおけるこれからの「データ整備」
shomaekawa
1
200
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
1k
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Scaling GitHub
holman
463
140k
Site-Speed That Sticks
csswizardry
11
880
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
GraphQLとの向き合い方2022年版
quramy
49
14k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Docker and Python
trallard
46
3.6k
Transcript
Wykorzystanie klastra Apache Mesos w deploymencie aplikacji pythonowych Kamil Warguła
1 github.com/quamilek
[email protected]
2
Apache Mesos 3
What is Mesos? Apache Mesos abstracts CPU, memory, storage, and
other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. 4
Mesos architecture Mesos master Standby master Standby master ZooKeeper quorum
Mesos slave Framework executor task Mesos slave Framework executor task Mesos slave Marathon executor task Hadoop executor task Framework scheduler 5
Resource allocation 6
Mesos UI 7
Mesos frameworks 8
Marathon 9
Marathon Marathon is an Apache Mesos framework for long-running applications
10
Marathon features • Web UI • JSON/REST API • HA
• Constraints • Health Checks • Event Subscription • Basic Auth and SSL 11
Resource utilization 12
Resource utilization 13
Mesos saves $$$ 14 old way mesos way
simple python REST app import falcon import json class ExampleResource:
def on_get(self, req, resp): data = {'foo': 'bar'} resp.body = json.dumps(data) api = falcon.API() api.add_route('/', ExampleResource()) 15
integration with gunicorn from gunicorn.app.base import BaseApplication class StandaloneApplication(BaseApplication): ...
def _get_config(): ... def main(): api = falcon.API() api.add_route('/', Example Resource()) config = _get_config() StandaloneApplication(api, config).run() if __name__ == '__main__': main() 16
How to run virtualenv env pip install falcon==0.3.0 gunicorn==19.3.0 source
env/bin/activate python api.py -p 8099 17
Pack requirements with PEX pex -v -r falcon==0.3.0 -r gunicorn==19.3.0
-o app.pex 18
How to run app with pex ./app.pex api.py -p 8099
19
DEMO 20
Marathon UI 21
Marathon app definition { "id": "appname", "cmd": "python my_app.py", "cpus":
0.5, "instances": 1, "mem": 64, "uris": [ "http://artifacts.local/my_app_0.1.0.tar.gz" ] } 22
Marathon - deploy python app DEMO 23
app configuration • environment variables • distributed configuration service (ZooKeeper,
ETCD) 24
Service Discovery • Marathon-Consul • Mesos-DNS 25
Load Balancing • haproxy-marathon-bridge 26
Scale app - manual import json import requests url =
'http://marathon.local/v2/apps/myapp' data = {'instances': 5} requests.put(url, data=json.dumps(data)) 27
Scale app - manual DEMO 28
Autoscale app MIN_INSTANCES_COUNT = 2 MAX_INSTANCES_COUNT = 20 if overloaded_instances
>= 0.9 * total_instances: total_instances += 2 if total_instances <= MAX_INSTANCES_COUNT: scale_app(total_instances) elif overloaded_instances <= 0.5 * total_instances: total_instances -= 2 if total_instances >= MIN_INSTANCES_COUNT: scale_app(total_instances) 29
Autoscale app DEMO 30
Thank you! 31 goo.gl/qWivrx
Q/A? 32 goo.gl/qWivrx