Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Experiences in Asynchronous Application Development with Crossbar.io by Adam Jorgensen

Pycon ZA
October 05, 2017

Experiences in Asynchronous Application Development with Crossbar.io by Adam Jorgensen

Recently at OpenDNA we've been rebuilding one of our core internal applications from the ground-up. Early on we decided that we wanted to make the new system completely asynchronous, leveraging the ability of modern browsers to communicate with the server in a bi-directional fashion unconstrained by the usual flow of a RESTful application.

There are a number of ways to implement this kind of application in Python but only one of them (in our humble opinion) is really worth investing time in: Crossbar.io

In this talk I will cover the process of developing a purely asynchronous application using Crossbar from top to bottom, detailing everything from high-level design considerations to low-level code decisions to potential pitfalls and how to avoid them.

The aim: To convince you that your next application should be built using Crossbar!

This talk should be suitable for any audience but will be of particular interest to developers looking for a way to leverage WebSockets and/or WAMP to develop highly or even purely asynchronous applications.

Pycon ZA

October 05, 2017
Tweet

More Decks by Pycon ZA

Other Decks in Programming

Transcript

  1. Who am I and what is ? • I’m Adam

    (you probably guessed that already though…) • Senior Backend Developer at OpenDNA • OpenDNA is a company leveraging Machine Learning and Python to • Perform User Behaviour and Smart Content Analyses • Provide Real-time Insights and Psychographic Profiles • We work primarily in Python 3 and leverage various AWS services • We’re hiring! Checkout out http://opendna.ai for more details… • As per ISO 476105, there is a good coffee machine!
  2. WAMP, Crossbar.io, Autobahn, Asyncio, fizz, buzz, etc! asyncio • Part

    of the Python 3.4+ standard library • Provides tools for writing single- threaded co-operative concurrent code…and more!
  3. HTTP Protocol vs. WAMP Protocol Feature HTTP Protocol WAMP Protocol

    State Stateless* Stateful Directionality Uni-directional Bi-directional Duplex Half-duplex Full-duplex Connection Transient**, optionally encrypted Persistent, optionally encrypted Message methods/verbs GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH Call, Register, Publish, Subscribe Message format Plain text JSON/MessagePack/CBOR Authentication Static/dynamic Basic access, Digest access, Custom challenge-response implemented in application Static/dynamic versions of Anonymous, Ticket, CRA, Cryptosign, Cookie, TLS Authorization Implemented in application Static/dynamic URI-based * State is usually implemented at the application level using various methods ** HTTP/1.1 provides a keep-alive mechanism which sees common usage
  4. Getting started with Crossbar and Autobahn • Crossbar/Autobahn Installation •

    Option 1: Docker • See https://crossbar.io/docs/Getting-Started/ • Option 2: Virtualenv • See https://crossbar.io/docs/Installation/ • Crossbar Project management • CLI tool • Node config file • WAMP Components • Easy to write • Running can be a bit more tricky… • asyncio components vs. twisted components: autobahn-python-runners to the rescue!
  5. Coming from a synchronous world… • asyncio library ecosystem is

    small but growing… • aio-libs organization on GitHub has clients for: HTTP, Redis, MySQL, PostgreSQL, ODBC, ZeroMQ, Kafka and more! • aioes is out and official elasticsearch-py-async is in  • ORM situation is tricky 1. sqlalchemy_aio exists but doesn’t allow usage of the ORM component 2. peewee-async provides Django-like Manager layer over peewee Option 2 is superior • No asyncio options? Don’t worry, there’s still hope! • Easy: Expensive sync call results can be cached for async access via aioredis • Less easy: Leverage asyncio.run_in_executor
  6. Framework not included… …but don’t worry, you won’t really miss

    it  • Compared to HTTP web frameworks, Crossbar and AutoBahn is quite minimal… • Autobahn-Python models each component as a class • Routed RPC takes the place of HTTP methods • PubSub has no direct comparison in HTTP web frameworks • No concept of request or response objects/thread-locals • Data supplied to end-points available via arguments and keyword arguments • Coercion of input/output data to and from Python types automatic • WSGI-style middleware replaced by decorators and/or inheritance • Everything else is up to you!
  7. Component-based Application Design • WAMP components • Are inherently modular

    • Have locational transparency • Can behave as both “clients” and “servers” to other WAMP components • In Python, implemented using classes • Do’s and Don’ts for Python components • Don’t store state on component class instances • Don’t interact with other component classes outside of RPC and PubSub • Don’t make blocking calls outside of __init__* • Don’t perform long-running work in end-points** • Do handle RPC call failures gracefully • Do use decorators to encapsulate common functionality * If you must, consider using the asyncio thread/process executor system
  8. Authentication and Authorization • Support baked in at Router level

    • Configured at Node level • Dynamic authenticators/authorizers implemented using components • Authentication end-point • Receives Realm and Authid • Responds with Role • Authorization end-point • Receives Session, URI, and Action • Responds with bool or dict • Overall a nice system • Highly flexible… • But with some limits to keep things clean 
  9. Debugging and Testing • Debugging • Initially seems difficult due

    to Crossbar being a process manager… • ….just run individual Components with PyDevD or normal breakpoints • Breakpoints on await…you can’t just step over them! • Interactive Testing • Initially tricky before our web front-end was up and running… • …so we developed a REPL Component! • OSS version: https://github.com/opn-oss/autobahn-python-repl • Automated Testing • pytest-asyncio • aiounittest
  10. Danger, Will Robinson! • Deadlocks possible in single-threaded code! Example:

    • Function A requests connection from aiomysql pool • Function A calls Function B • Function B requests connection from aiomysql pool • Fire off many calls to Function A, watch deadlock fireworks! • File I/O is blocking • Use aiofiles to get around this • peewee-async can accidentally make synchronous calls… • Calling a co-routine doesn’t automatically start it!
  11. Conclusion • Overall • Crossbar and Autobahn have been great

    to work with • Relative simplicity compared to HTTP frameworks is nice • Flexibility is also extremely impressive • asyncio ecosystem needs more love • Is Crossbar for you? • Bad fit for multi-page web applications • Good fit for single-page web applications and mobile app backends • Great fit for soft real-time applications and IoT • Bonus: Plays nicely in multi-language projects!