from Sync to Async Python, a AsyncIO migration

A tale about the migration of ZeroServices from python 2.7 / ØMQ / Tornado to python 3.4 / ØMQ / Asyncio

1. FROM SYNC TO ASYNC PYTHON, A ASYNCIO MIGRATION BORIS FELD - PYCONFR 2015
2. ZeroServices MicroServices framework Peer communication Peer discovery Real-time notifications
3. Asynchronous Synchronous -> process stuck during IO Asynchronous -> process could do something else during IO
4. V0
5. No tests Proof of concept validate the feasibility. Small enough. Tests aren’t worth it.
6. V0 ZMQ communication Tornado integration with pyzmq Not WSGI: Request (client) -> Response (server) Doesn’t works well with web socket, etc…
7. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
8. Pyzmq tips ZmqStream are your friends. stream_sub = zmqstream.ZMQStream(socket_sub) stream_sub.on_recv(process_message)
9. Result Works! Validate the design / architecture. Some parts were asynchronous.
10. Why changes? Want to test python 3.4 because it’s the future. Want to use asyncio because of the highlight.
11. V1
12. V1 PYTHON 3!
13. Python 3.4 migration Mostly done thanks to 2to3. Dependencies were migrated. No six, drop support to python2.
14. Python 3 <3 Str/bytes separation
15. Python 2 Mix of unicode and str, conversion when UnicodeEncodeError and UnicodeDecodeError.
16. Environnement - Bytes Str / Bytes Application Str
17. Str / Bytes Everything from outside is bytes and needs to be converted to str. Everything to outside is str and needs to be converted to bytes.
18. Synchronous Still synchronous design: Easier to think about Easier to reason about design Small design tuning Tests!
19. TDD rewriting Redesign using TDD Add test, copy « good » V0 code, refactoring.
20. Dependency injection Make IO parts interchangeable with the same API. So your tests could use in-memory stubs.
21. Interface testing class _BaseMediumTestCase(TestCase): def setUp(self): self.medium_1 = self.get_medium() self.medium_2 = self.get_medium() def test_send_response(self): return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
22. One test case per implementation class MemoryMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium = MemoryMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): def get_medium(self): medium = ZeroMQMedium(discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) medium.start() return medium
23. What is better ? Design more clean, clear separation of concerns. Modern Python. Tests \o/
24. Why changes? Because still synchronous.
25. V2
26. Asyncio Young but promising community Lots of libraries: aiozmq aiohttp
27. Asynchronous migration Easy to do with: Clean design Tests
28. Yield from Basically changes: for i in f(): yield i yield from f()
29. Tornado example in asyncio @asyncio.coroutine def handle(request): result = yield from fetch() return web.Response(body=result) def fetch(): response = yield from aiohttp.request('GET', 'http://ifconfig.me') return (yield from response.read())
30. Tornado example class GenAsyncHandler(RequestHandler): @gen.coroutine def get(self): result = yield self.fetch() self.write(result) @gen.coroutine def fetch(self): http_client = AsyncHTTPClient() response = yield http_client.fetch("http://ifconfig.me") raise Return(response.body)
31. Yield from Avoid the stack of gen.coroutine or gen.Task Replace by asyncio.coroutine async / await python 3.5 ?
32. Tests Already has some tests. Tests are your best friend
33. Tests Rewrite tests in asynchronous with this snippet: def _async_test(f): @wraps(f) def wrapper(self, *args, **kwargs): if not self.loop.is_running(): coro = asyncio.coroutine(f) future = coro(self, *args, **kwargs) self.loop.run_until_complete(asyncio.wait_for(future, 2, loop=self.loop)) else: return f(self, *args, **kwargs) return wrapper
34. Test migration class _BaseMediumTestCase(TestCase): def setUp(self): self.loop = asyncio.new_event_loop() asyncio.set_event_loop(None) self.medium_1 = self.loop.run_until_complete(self.get_medium(self.loop)) self.medium_2 = self.loop.run_until_complete(self.get_medium(self.loop)) @_async_test def test_send_response(self): yield from asyncio.sleep(0.1, loop=self.loop) return_value = {'data': 'ReturnValue'} message_type = 'MYMESSAGETYPE' message = {'foo': 'bar'} self.medium_2.service.on_message_mock.return_value = return_value result = yield from self.medium_1.send(self.medium_2.node_id, message, message_type=message_type) self.assertEqual(result, return_value)
35. Test migration II class MemoryMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium = MemoryMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium class ZeroMQMediumTestCase(_BaseMediumTestCase): @asyncio.coroutine def get_medium(self, loop): medium = ZeroMQMedium(loop=loop, discovery_class=MemoryDiscoveryMedium) medium.set_service(TestService('test_service', medium)) yield from medium.start() return medium
36. Tip Asyncio queue are a pain in the ass to test You want to send a query and when the yield from respond back, everything is fine.
37. Queue class ZmqDealerProtocol(aiozmq.ZmqProtocol): def msg_received(self, msg): self.queue.put_nowait(msg) while True: msg = yield from queue.get()
38. VS class ServerProtocol(object): def msg_received(self, msg): asyncio.async(self.callback(msg), loop=self.loop)
39. Refactoring Little refactoring on how parts interact which each other. Which is doing asynchronous and which doesn’t.
40. Asynchronous Good for scalability, not raw performances. Overhead worth it at scale.
41. Conclusion Python 3.4 migration is not so hard Asyncio migration is not so hard With clear design And tests And asynchronous mindset