Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Eventlet, ZeroMQ, and You
Search
Andrew Godwin
October 22, 2011
Programming
6
780
Eventlet, ZeroMQ, and You
A talk I gave at PyCon Ukraine 2011.
Andrew Godwin
October 22, 2011
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
240
Django Through The Years
andrewgodwin
0
130
Writing Maintainable Software At Scale
andrewgodwin
0
370
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
280
Async, Python, and the Future
andrewgodwin
2
580
How To Break Django: With Async
andrewgodwin
1
630
Taking Django's ORM Async
andrewgodwin
0
640
The Long Road To Asynchrony
andrewgodwin
0
570
The Scientist & The Engineer
andrewgodwin
1
660
Other Decks in Programming
See All in Programming
Flutterアプリを生成AIで生成する勘所
rizumita
0
230
なぜアジャイルがうまくいかないのか?
yum3
2
140
WEBアプリケーションにおけるAWS Lambdaを用いた大規模な非同期処理の実践
delhi09
PRO
7
3.4k
PHPを書く理由、PHPを書いていて良い理由 / Reasons to write PHP and why it is good to write PHP
seike460
PRO
5
270
ファーストペンギンBot @Qiita Hackathon 2024 予選
dyson_web
0
200
DroidKaigi 2024 - You will 💜 Becoming a fullstack developer with your Android skills
brah
0
110
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
5
170
型付きで行うVSCode拡張機能開発 / VSCode Meetup #31
mazrean
0
170
Progressive Web Apps for Rails developers
siaw23
2
510
CSC509 Lecture 02
javiergs
PRO
0
140
Infrastructure as Code でセキュリティを楽にしよう!
konokenj
5
1.4k
What is TDD?
urakawa_jinsei
1
190
Featured
See All Featured
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.1k
Debugging Ruby Performance
tmm1
72
12k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
Why Our Code Smells
bkeepers
PRO
334
57k
Faster Mobile Websites
deanohume
304
30k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Building Better People: How to give real-time feedback that sticks.
wjessup
359
19k
Bash Introduction
62gerente
608
210k
Robots, Beer and Maslow
schacon
PRO
157
8.2k
KATA
mclloyd
27
13k
The Illustrated Children's Guide to Kubernetes
chrisshort
47
48k
What the flash - Photography Introduction
edds
67
11k
Transcript
Eventlet, ZeroMQ & You Andrew Godwin @andrewgodwin
Who am I? http://www.flickr.com/photos/maistora/3014414972
The Plan http://www.flickr.com/photos/snapsi42/2609025344/
Eventlet Asynchronous concurrency framework http://www.flickr.com/photos/pagedooley/2814895287/
import eventlet from eventlet.green import urllib2 urls = ['http://ep.io', 'http://t.co']
results = [] def fetch(url): results.append(urllib2.urlopen(url).read()) for url in urls: eventlet.spawn(fetch, url)
Cooperative Threading You have to yield control. http://www.flickr.com/photos/nickwheeleroz/3406931272/
import eventlet import time def thread1(): time.sleep(5) # Does not
yield print 'five seconds a' eventlet.spawn(thread1) time.sleep(5) # Does not yield print 'five seconds b'
import eventlet import time def thread1(): eventlet.sleep(5) print 'five seconds
a' eventlet.spawn(thread1) eventlet.sleep(5) print 'five seconds b'
Greening the World It's that or monkeypatching. http://www.flickr.com/photos/mightyboybrian/3457507731/
import urllib2 from eventlet.green import urllib2 import zmq from eventlet.green
import zmq import socket from eventlet.green import socket
Nice Primitives Not to mention less race conditions. http://www.flickr.com/photos/mightyboybrian/3457507731/
from eventlet.timeout import Timeout try: with Timeout(30): do_stuff() except Timeout:
abort_abort()
from eventlet.semaphore import Semaphore sem = Semaphore(1) with sem: use_resource()
from eventlet.queue import Queue q = Queue() def worker(): while
True: print q.get() def producer(): while True: eventlet.sleep(1) q.put('lol') eventlet.spawn(worker) eventlet.spawn(producer)
from eventlet import GreenPool from eventlet.pools import Pool conns =
Pool(create=make_db_connection) threads = GreenPool(10) def querier(id): with pool.item() as conn: conn.execute('... WHERE id = %s', [id]) pool.imap(querier, [1, 2, 3, 4, 5])
Experiences from production There's still race conditions and bad libraries.
http://www.flickr.com/photos/iamdabe/4931554963/
Case Study: Epio Loadbalancer Now at version three. http://www.flickr.com/photos/jasohill/118616905/
Version One: HAProxy + reloader Occasionally slow, occasionally didn't reload.
Version Two: Custom Integrated Worked alright, but leaked sockets.
Version Three: Fresh, Separate Codenamed ”Mantrid”
address = ('0.0.0.0', 80) family = socket.AF_INET sock = eventlet.listen(address,
family) eventlet.serve( sock, self.handle, concurrency = 10000, )
address = ('127.0.0.1', 8042) family = socket.AF_INET sock = eventlet.listen(address,
family) management_app = ManagementApp(self) wsgi.server( sock, management_app.handle, )
class Spin(Action): def handle(self, sock, read_data, path, headers): while True:
# Sleep first eventlet.sleep(self.check_interval) # Check for another action action = self.balancer.resolve_host(self.host) if not isinstance(action, Spin): return action.handle( sock, read_data, path, headers )
ZeroMQ I was too lazy to use an Ø. http://www.flickr.com/photos/andresrueda/2926348197/
Not a message queue More like 'better sockets'. http://www.flickr.com/photos/96dpi/3816391053/
REQ / REP PUB / SUB PUSH / PULL
from eventlet.green import zmq ctx = zmq.Context() sock = ctx.socket(zmq.REQ)
sock.connect('tcp://127.0.0.1:1234') sock.connect('tcp://127.0.0.1:4321') sock.send('hi there') print sock.recv()
from eventlet.green import zmq ctx = zmq.Context() sock = ctx.socket(zmq.REP)
sock.bind('tcp://0.0.0.0:1234') while True: message = sock.recv() sock.send('you said: %s' % message)
from eventlet.green import zmq ctx = zmq.Context() sock = ctx.socket(zmq.PUB)
sock.connect('tcp://127.0.0.1:1234') sock.send('Hello, world!')
from eventlet.green import zmq ctx = zmq.Context() sock = ctx.socket(zmq.SUB)
sock.bind('tcp://0.0.0.0:1234') while True: print sock.recv()
The restrictions It's not quite too good to be true.
http://www.flickr.com/photos/maistora/3237164755/
Advanced ZeroMQ & Eventlet When one greenthread is not enough
http://www.flickr.com/photos/maistora/3237164755/
sock = ctx.socket(zmq.REQ) sock.send('help me! you're my only hope!') try:
with Timeout(30): response = sock.recv() deal_with(response) except Timeout: cleanup()
@zmq_loop(zmq.XREP, 8000) def handle_request(string): eventlet.sleep(10) return string * 2
try: parts = sock.recv_multipart(flags=zmq.NOBLOCK) except zmq.ZMQError, e: if e.errno ==
zmq.EAGAIN: eventlet.sleep(0.1) continue else: raise identities = parts[:parts.index("")] def call(identities, data): result = func(self, data) sock.send_multipart(identities + ["", result]) eventlet.spawn_n(call, identities, parts[-1])
Case Study: Epio Logging We really, really like data http://www.flickr.com/photos/jasohill/118616905/
Logging clients They're the things that make logs
Logging servers They're the things that save and store logs
# Servers zmq.PULL -> Recieves logs zmq.REP -> Lets website
query for log data # Clients zmq.PUSH -> Sends logs, connected to all servers
Conclusions Where Andrew waffles for a bit. http://www.flickr.com/photos/oimax/108058706/
Thanks. Andrew Godwin @andrewgodwin http://aeracode.org