Upgrade to Pro — share decks privately, control downloads, hide ads and more …

from Sync to Async Python, a AsyncIO migration

410e3353165c33043ab69be7fc366428?s=47 Boris Feld
October 17, 2015

from Sync to Async Python, a AsyncIO migration

A tale about the migration of ZeroServices from python 2.7 / ØMQ / Tornado to python 3.4 / ØMQ / Asyncio

410e3353165c33043ab69be7fc366428?s=128

Boris Feld

October 17, 2015
Tweet

Transcript

  1. FROM SYNC TO ASYNC PYTHON, A ASYNCIO MIGRATION BORIS FELD

    - PYCONFR 2015
  2. ZeroServices MicroServices framework Peer communication Peer discovery Real-time notifications

  3. Asynchronous Synchronous -> process stuck during IO Asynchronous -> process

    could do something else during IO
  4. V0

  5. No tests Proof of concept validate the feasibility. Small enough.

    Tests aren’t worth it.
  6. V0 ZMQ communication Tornado integration with pyzmq Not WSGI: Request

    (client) -> Response (server) Doesn’t works well with web socket, etc…
  7. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield

    self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
  8. Pyzmq tips ZmqStream are your friends. stream_sub = zmqstream.ZMQStream(socket_sub) stream_sub.on_recv(process_message)

  9. Result Works! Validate the design / architecture. Some parts were

    asynchronous.
  10. Why changes? Want to test python 3.4 because it’s the

    future. Want to use asyncio because of the highlight.
  11. V1

  12. V1 PYTHON 3!

  13. Python 3.4 migration Mostly done thanks to 2to3. Dependencies were

    migrated. No six, drop support to python2.
  14. Python 3 <3 Str/bytes separation

  15. Python 2 Mix of unicode and str, conversion when UnicodeEncodeError

    and UnicodeDecodeError.
  16. Environnement - Bytes Str / Bytes Application Str

  17. Str / Bytes Everything from outside is bytes and needs

    to be converted to str. Everything to outside is str and needs to be converted to bytes.
  18. Synchronous Still synchronous design: Easier to think about Easier to

    reason about design Small design tuning Tests!
  19. TDD rewriting Redesign using TDD Add test, copy « good

    » V0 code, refactoring.
  20. Dependency injection Make IO parts interchangeable with the same API.

    So your tests could use in-memory stubs.
  21. Interface testing class _BaseMediumTestCase(TestCase): def setUp(self): self.medium_1 = self.get_medium() self.medium_2

    = self.get_medium() def test_send_response(self): return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
  22. One test case per implementation class MemoryMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium

    = MemoryMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium = ZeroMQMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium
  23. What is better ? Design more clean, clear separation of

    concerns. Modern Python. Tests \o/
  24. Why changes? Because still synchronous.

  25. V2

  26. Asyncio Young but promising community Lots of libraries: aiozmq aiohttp

  27. Asynchronous migration Easy to do with: Clean design Tests

  28. Yield from Basically changes: for i in f(): yield i

    yield from f()
  29. Tornado example in asyncio @asyncio.coroutine def handle(request): result = yield

    from fetch() return web.Response(body=result) def fetch(): response = yield from aiohttp.request('GET', 'http://ifconfig.me') return (yield from response.read())
  30. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield

    self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
  31. Yield from Avoid the stack of gen.coroutine or gen.Task Replace

    by asyncio.coroutine async / await python 3.5 ?
  32. Tests Already has some tests. Tests are your best friend

  33. Tests Rewrite tests in asynchronous with this snippet: def _async_test(f):

    @wraps(f) def wrapper(self, *args, **kwargs): if not self.loop.is_running(): coro = asyncio.coroutine(f) future = coro(self, *args, **kwargs) self.loop.run_until_complete(asyncio.wait_for(future, 2, loop=self.loop)) else: return f(self, *args, **kwargs) return wrapper
  34. Test migration class _BaseMediumTestCase(TestCase): def setUp(self): self.loop = asyncio.new_event_loop() asyncio.set_event_loop(None)

    self.medium_1 = self.loop.run_until_complete(self.get_medium(self.loop)) self.medium_2 = self.loop.run_until_complete(self.get_medium(self.loop)) @_async_test def test_send_response(self): yield from asyncio.sleep(0.1, loop=self.loop) return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = yield from self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
  35. Test migration II class MemoryMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium

    = MemoryMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium = ZeroMQMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium
  36. Tip Asyncio queue are a pain in the ass to

    test You want to send a query and when the yield from respond back, everything is fine.
  37. Queue class ZmqDealerProtocol(aiozmq.ZmqProtocol): def msg_received(self, msg): self.queue.put_nowait(msg) while True: msg

    = yield from queue.get()
  38. VS class ServerProtocol(object): def msg_received(self, msg): asyncio.async(self.callback(msg), loop=self.loop)

  39. Refactoring Little refactoring on how parts interact which each other.

    Which is doing asynchronous and which doesn’t.
  40. Asynchronous Good for scalability, not raw performances. Overhead worth it

    at scale.
  41. Conclusion Python 3.4 migration is not so hard Asyncio migration

    is not so hard With clear design And tests And asynchronous mindset