Distributed Applications in Python: Celery vs Crossbar by Adam Jorgensen

In this talk I will discuss two specific methods of implementing distributed applications in Python. Distributed applications allow one to improve resiliency and performance, although this can come at the cost of increased complexity. The trick is to apply a distributed application framework in situations where that complexity is less significant than the benefits it provides.

Of the two systems I will be covering, Celery is by far the better known and more mature, with version 0.1.0 having been released in 2009. Celery is currently at version 3.1.23 and has come a long way in that time. Today it provides a stable and mature distributed task queue with a focus on real-time execution although it is also capable of cron-like scheduled operations. Celery is extremely flexible and configurable, although this comes at the cost of some complexity. Thankfully the documentation is rock solid and the community large.

Crossbar is a newer project that was first released in 2013. Compared to Celery it's less well known, has a smaller community and the documentation is definitely lacking in terms of quality and robustness. Despite these flaws Crossbar is a very exciting project that is very pleasant to work with once you've gotten past the initial learning curve. While Celery provides a distributed task queue, Crossbar functions as a WAMP router. WAMP is a routed messaging protocol built on WebSockets that provides RPC and PubSub. As such it targets a slightly different space from Celery. That said, there is enough overlap for discussion.

Picking a distributed application framework can be tricky as there are a variety of options available. My aim with this talk is to compare and contrast two specific frameworks that I have some experience with, illustrating their similarities, differences, strengths and weaknesses. I will cover some basic examples for each framework and hopefully provide the audience with a better idea of why they might choose to use one or the other (or even both!).

There is no specific audience focus for this talk as I imagine distributed applications can be of benefit to many different use-cases. With that said, my background is in web development and hence my discussion of the two frameworks in question will probably draw on that experience. Regardless, I think that anyone with an interest in distributed applications could benefit from this talk.

Transcript

1. Distributed Applications in Python: Celery vs. Crossbar (I couldn’t find a single “cute” image for Crossbar ) By Adam Jorgensen, https://github.com/oopman/
2. • Provide online bookings for medical appointments • Integration with PMA software via our API • Our tech stack includes:
3. Distributed Computing… And that’s just the tip of the iceberg, it’s an extremely broad topic!
4. Distributed Computing in Python…
5. More Distributed Computing in Python!
6. Enough content for a million talks… …so let’s skip most of it and just talk about two players in this game: Celery Crossbar
7. • Celery Applications consist primarily of Tasks from myapp.celery import app @app.task def add(x, y): return x + y Celery: A Distributed Task Queue • Tasks can be executed synchronously or asynchronously from myapp.tasks import add # Synchronous calls occur in the current thread/process add(1,2) # Asynchronous calls result in execution by a Celery Worker add.delay(1,2) # Short form for basic usage add.apply_async(args=[1], kwargs={'y': 2}) # Long form allows for additional execution options • Celery Deployment • 1 or more Worker instances • Optional: 1 Beat instance • Optional: Flower
8. Celery: The Crunchy Parts • Workflow system • Signatures • Callbacks • Chains • Groups • Chords • Map • Chunks • Task routing • Signals • Django integration
9. Celery: The Good Parts • Fairly mature • Highly configurable • Easy to scale • Celery Beat • Documentation
10. Celery: The Bad Parts • Highly configurable • Prone to peculiar issues • Debugging
11. Crossbar: Networking for Applications • Crossbar is a WAMP router • WAMP? WebSockets? Autobahn? from autobahn.twisted.util import sleep from twisted.internet.defer import inlineCallbacks from autobahn import wamp from autobahn.twisted.wamp import ApplicationSession class ExamplePython2Component(ApplicationSession): @wamp.register(u'com.demo.add') def add(x, y) return x + y @inlineCallbacks def onJoin(self, details): yield self.register(self) counter = 0 while True: self.publish('com.demo.tick', counter) counter += 1 yield sleep(1) from asyncio import sleep from autobahn import wamp from autobahn.asyncio.wamp import ApplicationSession class ExamplePython3Component(ApplicationSession): @wamp.register(u'com.demo.add') def add(x, y) return x + y async def onJoin(self, details): await self.register(self) counter = 0 while True: self.publish('com.demo.tick', counter) counter += 1 await sleep(1)
12. Crossbar: The Good Parts • Impressive connection handling abilities • Components are location agnostic • Component hosting • Extensible security system • Shared registrations • Polyglot friendly • Embedded HTTP server
13. Crossbar: The Bad Parts • Twisted requires usage of alternate ecosystem of libraries • No Clustering support. Yet… • Documentation is out of date
14. Celery vs. Crossbar Reasons to use Celery: • Horizontal/Vertical Scalability • Scheduling • Workflows Reasons to use Crossbar: • Non-blocking asynchronous paradigm • Authentication/Authorization • WAMP