Upgrade to Pro — share decks privately, control downloads, hide ads and more …

from Sync to Async Python, a AsyncIO migration

Boris Feld
October 17, 2015

from Sync to Async Python, a AsyncIO migration

A tale about the migration of ZeroServices from python 2.7 / ØMQ / Tornado to python 3.4 / ØMQ / Asyncio

Boris Feld

October 17, 2015
Tweet

More Decks by Boris Feld

Other Decks in Programming

Transcript

  1. V0

  2. V0 ZMQ communication Tornado integration with pyzmq Not WSGI: Request

    (client) -> Response (server) Doesn’t works well with web socket, etc…
  3. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield

    self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
  4. Why changes? Want to test python 3.4 because it’s the

    future. Want to use asyncio because of the highlight.
  5. V1

  6. Python 3.4 migration Mostly done thanks to 2to3. Dependencies were

    migrated. No six, drop support to python2.
  7. Str / Bytes Everything from outside is bytes and needs

    to be converted to str. Everything to outside is str and needs to be converted to bytes.
  8. Synchronous Still synchronous design: Easier to think about Easier to

    reason about design Small design tuning Tests!
  9. Interface testing class _BaseMediumTestCase(TestCase): def setUp(self): self.medium_1 = self.get_medium() self.medium_2

    = self.get_medium() def test_send_response(self): return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
  10. One test case per implementation class MemoryMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium

    = MemoryMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium = ZeroMQMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium
  11. What is better ? Design more clean, clear separation of

    concerns. Modern Python. Tests \o/
  12. V2

  13. Tornado example in asyncio @asyncio.coroutine def handle(request): result = yield

    from fetch() return web.Response(body=result) def fetch(): response = yield from aiohttp.request('GET', 'http://ifconfig.me') return (yield from response.read())
  14. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield

    self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
  15. Yield from Avoid the stack of gen.coroutine or gen.Task Replace

    by asyncio.coroutine async / await python 3.5 ?
  16. Tests Rewrite tests in asynchronous with this snippet: def _async_test(f):

    @wraps(f) def wrapper(self, *args, **kwargs): if not self.loop.is_running(): coro = asyncio.coroutine(f) future = coro(self, *args, **kwargs) self.loop.run_until_complete(asyncio.wait_for(future, 2, loop=self.loop)) else: return f(self, *args, **kwargs) return wrapper
  17. Test migration class _BaseMediumTestCase(TestCase): def setUp(self): self.loop = asyncio.new_event_loop() asyncio.set_event_loop(None)

    self.medium_1 = self.loop.run_until_complete(self.get_medium(self.loop)) self.medium_2 = self.loop.run_until_complete(self.get_medium(self.loop)) @_async_test def test_send_response(self): yield from asyncio.sleep(0.1, loop=self.loop) return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = yield from self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
  18. Test migration II class MemoryMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium

    = MemoryMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium = ZeroMQMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium
  19. Tip Asyncio queue are a pain in the ass to

    test You want to send a query and when the yield from respond back, everything is fine.
  20. Refactoring Little refactoring on how parts interact which each other.

    Which is doing asynchronous and which doesn’t.
  21. Conclusion Python 3.4 migration is not so hard Asyncio migration

    is not so hard With clear design And tests And asynchronous mindset