Share
Embed
Introduction to Gevent
A gentle introduction to Gevent (http://gevent.org/)
Isman Firmansyah
August 31, 2013
Transcript
38 slides
-
WHAT IS GEVENT gevent is a -based Python networking library
that uses to provide a high-level synchronous API on top of the event loop. coroutine greenlet libevent -
COROUTINE TL;DR http://en.wikipedia.org/wiki/Coroutine or in a nutshell: Well, a coroutine
is a subroutine or function that can pause in the middle and return a value. When you call it again it picks up right where it was when you paused, environment intact, and keeps on going. http://is.gd/J2Zq0R -
WHEN TO USE GEVENT THE CONCLUSION (SO FAR) Use gevent
for concurrent execution!! -
CONCURRENCY DOES MATTER Concurrency is the composition of independently executing
computations. It simulates the real-world. → Optimize performance and scalability. → http://is.gd/G0ewD8 http://is.gd/IsYrKF -
FOR WEB DEVELOPERS: TACKLE THE C10K PROBLEM http://www.kegel.com/c10k.html Handle ten
thousand clients simultaneously!! -
THREADING GIL problems!! Might requires locking mechanism CPU-bound programs run
significantly worse -
CONCURRENT.FUTURES Mimic of Java's concurrent.futures Available in Python 3.2+ Backport
for Python 2.5 - 3.1 → https://pypi.python.org/pypi/futures -
TWISTED http://twistedmatrix.com/ OH: "Twisted is heavy, I agree but not
bloated." uses Twisted as its core Graphite -
TORNADO http://www.tornadoweb.org/ Battle-tested (originally developed at FriendFeed) Avoid libraries which
doesn't compatible with Tornado's IOLoop -
WHY NOT X - - - - - ... PyPy
NodeJS Erlang Elixir Go Let's talk after this session ends :) -
SYNCHRONOUS API Write program that looks blocking, but actually runs
asynchronously. # blocking for x in xrange(1, 11): time.sleep() # non-blocking ops = [] for x in xrange(1, 11): ops.append(gevent.spawn(time.sleep)) gevent.joinall(ops) -
NO CALLBACK-WORRIES Forget the callback!! def _collection_handler(self, callback=None, on_error=None): def
inner(fd, ev): try: self._db.read_query_result() resultset = self._db.use_result() rows = self._get_rows(resultset) if callback: callback(rows) except Exception, exc: self._throw_error(exc, on_error) finally: self._ioloop.remove_handler(self._db.fd) return inner -
MONKEY PATCHING UTILITY Patches stdlib and 3rd-party modules to become
cooperative. import gevent.monkey gevent.monkey.patch_all() ops = [] for x in xrange(1, 11): ops.append(gevent.spawn(time.sleep)) gevent.joinall(ops) Caveats: cannot be used for C-extension -
HUGE COLLECTIONS OF COMPATIBLE LIBRARIES Thanks to monkey patching, we
have abundant async libraries: MySQL => pymysql/ultramysql Redis => redis-py Beanstalk => beanstalkc Postgres => psycopg (with specific setup) ... and more -
TOOLS BUILT ON TOP/SUPPORTS GEVENT / : load testing :
Python implementation of the Socket.IO : Requests + Gevent : WSGI server ... and more locust.io boom gevent-socketio grequests gunicorn -
SPAWNING GREENLET import gevent from gevent import Greenlet def foo(message,
n): gevent.sleep(n) print message # Initialize a new Greenlet instance running the named function foo thread1 = Greenlet.spawn(foo, "Hello", 1) # Wrapper for Greenlet.spawn thread2 = gevent.spawn(foo, "I live!", 2) threads = [thread1, thread2] # Block until all threads complete. gevent.joinall(threads) -
MONITORING GREENLET Use Greenlet states: started, ready, successful(), value, and
exception. Mostly, you'll only need value and exception. -
RETRIEVING THE VALUE FROM A GREENLET Use greenlet.value: import gevent
def win(): return 'You win!' if __name__ == "__main__": winner = gevent.spawn(win) gevent.joinall([winner]) print "winner value:", winner.value # "You win!" -
HANDLING EXCEPTION Use greenlet.exception: import gevent def fail(): raise Exception('You
fail at failing.') if __name__ == "__main__": loser = gevent.spawn(fail) # Exceptions raised in the Greenlet, stay inside the Greenlet. try: gevent.joinall([loser]) except Exception as exc: print "This will never be reached!" # It is possible though to raise the exception again outside #raise loser.exception # Or using #loser.get() # A stack trace will be printed to stdout but it # will not unwind the stack of the parent. print "loser exception:", loser.exception -
WITHOUT GEVENT import time def boring(): print "without gevent" for
x in xrange(1, 11): print "boring!", x time.sleep(1) if __name__ == "__main__": boring() -
WITH GEVENT import gevent.monkey gevent.monkey.patch_all() import time def green(): print
"with gevent" ops = [] for x in xrange(1, 11): print "boring!", x ops.append(gevent.spawn(time.sleep)) gevent.joinall(ops) if __name__ == "__main__": green()