A Plug for Plugins (PyGotham 2017)

The philosophy of small, focused software has its merits, but for most of us, software is big. From your browser to your kernel, the size of these applications is lost on most of us. How do these programs even grow to this size?

In all of these cases, the answer involves an often-overlooked pattern: the plugin. Call them modules or extensions, if you’d prefer, but they are wildly successful. In fact, the only thing wider than the success of plugin-based architecture is the variety of implementations, especially in Python’s dynamic environment.

This talk covers the basis for plugins, reviews Python’s current offerings using examples, and provides guidance as to the bright future of plugin architecture.

Transcript

1. A Plug for Plugins PyGotham 2017 Mahmoud Hashemi
2. Plugins Add-ons, components, modules, themes, widgets, extensions
3. Plugins Pieces of software, Loaded by a core program, To add behaviors. (a haiku)
4. Brand issues with plugins Functional, but not very dignified.
5. Brand issues with plugins Tired, possibly a code smell?
6. Brand issues with plugins Or maybe why WordPress breaks so much? Tired, possibly a code smell?
7. Brand issues with plugins Eclipsed by some very bad experiences.
8. Plugins everywhere! □ Browser extensions □ Kernel modules □ Shells (.bashrc & .bash_profile) □ Good editors (from Sublime to emacs) □ So many more! When plugins work, you might not even notice.
9. Plugins! Up next: □ Why plugins exist □ Reasons to use plugins □ Using plugins in Python applications
10. Stages of development skills □ Sticking values together >>> 2 + 2 □ Sticking lines together □ Sticking functions together □ Sticking modules together □ Sticking behaviors together (Unsticking those modules & functions)
11. When to use plugin architecture □ Reduced core size □ Work around licensing issues □ Community development ▪ Establish an API for code integration ▪ Especially for applications
12. Case study: conda cli Simple plugins, used internally
13. conda The Python-first, cross-platform package manager. (Imagine if pip and virtualenv had more-powerful fusion form)
14. conda internals Inside of https://github.com/conda/conda:
15. conda install plugin cli/main_install.py:
16. conda core cli/main.py:
17. Pros and cons of conda’s plugins □ Pros ▪ Simple ▪ Dynamic loading of hooks ▪ Only requires standard library □ Cons ▪ Not systemized (copyable but not general) ▪ Not user-facing
18. Case study: pluginbase Popular reusable plugin system
19. pluginbase From the makers of Flask and Click, 3-step plugins: https://github.com/pluginbase/pluginbase
20. Pros and cons of pluginbase □ Pros ▪ Generalized ▪ User-facing □ Cons ▪ No distribution story ▪ Path wrangling ▪ Attempts to: ➢ “Localize” modules ➢ Enable simultaneous loading of multiple versions of plugins
21. Let’s talk about modules □ Python’s built-in plugin system □ Simple, but also not so much ▪ PEP 302 Finders and Loaders □ Global by design ▪ sys.modules
22. Case study: gather Shiny new entrypoints-based system
23. First: Entry Points One of many setuptools features: “Entry points are a simple way for distributions to ‘advertise’ Python functions for use by other packages.” For instance: import pkg_resources for entry_point in pkg_resources.iter_entry_points('label'): registry[entry_point.name] = entry_point.load() http://amir.rachum.com/blog/2017/07/28/python-entry-points/
24. gather First, we need an application. core_app.py:
25. gather Now we need to build a plugin. core_app_plugin.py: (We celebrate usefulness.)
26. gather And now the gather entry point special sauce:
27. gather And once we pip install everything, it’s time to celebrate:
28. Pros and cons of gather □ Pros ▪ Uses standard setuptools ▪ Developer-facing ▪ Distribution through pip ▪ Plugins remain independently usable/testable □ Cons ▪ Only user-facing if your users are developers
29. Other systems and use cases □ stevedore □ twisted.plugin □ Mercurial extensions □ pytest plugins (pluggy) □ venusian □ straight.plugin □ pylint plugins □ flake8 plugins □ zope.component □ Django ▪ Apps ▪ Middlewares ▪ Command extensions □ SQLAlchemy dialects/DBAPIs □ Sphinx extensions □ Buildout extensions □ Pike □ Dectate and Reg
30. Wrapping up We learned that plugins are: □ Redistributable components for composing behavior □ The architecture for scaling development ▪ Internal / External ▪ Open / Closed-source □ Not all plugin implementations are the same. □ Still plenty of room for innovation!
31. Questions? @mhashemi github.com/mahmoud sedimental.org/plugin_systems.html ?