The philosophy of small, focused software has its merits, but for most of us, software is big. From your browser to your kernel, the size of these applications is lost on most of us. How do these programs even grow to this size?
In all of these cases, the answer involves an often-overlooked pattern: the plugin. Call them modules or extensions, if you’d prefer, but they are wildly successful. In fact, the only thing wider than the success of plugin-based architecture is the variety of implementations, especially in Python’s dynamic environment.
This talk covers the basis for plugins, reviews Python’s current offerings using examples, and provides guidance as to the bright future of plugin architecture.
modules, themes, widgets,
Pieces of software,
Loaded by a core program,
To add behaviors.
Functional, but not very dignified.
Tired, possibly a code smell?
Or maybe why WordPress breaks so much?
Tired, possibly a code smell?
□ Browser extensions
□ Kernel modules
□ Shells (.bashrc & .bash_profile)
□ Good editors (from Sublime to emacs)
□ So many more!
When plugins work, you might not even notice.
□ Why plugins exist
□ Reasons to use plugins
□ Using plugins in Python applications
Stages of development skills
□ Sticking values together
>>> 2 + 2
□ Sticking lines together
□ Sticking functions together
□ Sticking modules together
□ Sticking behaviors together
(Unsticking those modules & functions)
When to use plugin architecture
□ Reduced core size
□ Work around licensing issues
□ Community development
■ Establish an API for code integration
■ Especially for applications
Case study: conda cli
Simple plugins, used internally
The Python-first, cross-platform package manager.
(Imagine if pip and virtualenv had more-powerful fusion form)
Inside of https://github.com/conda/conda:
conda install plugin
Pros and cons of conda’s plugins
■ Dynamic loading of hooks
■ Only requires standard library
■ Not systemized (copyable but not general)
■ Not user-facing
Case study: pluginbase
Popular reusable plugin system
From the makers of Flask and Click, 3-step plugins:
Pros and cons of pluginbase
■ No distribution story
■ Path wrangling
■ Attempts to:
➢ “Localize” modules
➢ Enable simultaneous loading of
multiple versions of plugins
Let’s talk about modules
□ Python’s built-in plugin system
□ Simple, but also not so much
■ PEP 302 Finders and Loaders
□ Global by design
Case study: gather
Shiny new entrypoints-based system
First: Entry Points
One of many setuptools features:
“Entry points are a simple way for distributions to
‘advertise’ Python functions for use by other packages.”
for entry_point in pkg_resources.iter_entry_points('label'):
registry[entry_point.name] = entry_point.load()
First, we need an application.
Now we need to build a plugin.
(We celebrate usefulness.)
And now the gather entry point special sauce:
And once we pip install everything, it’s time to celebrate:
Pros and cons of gather
■ Uses standard setuptools
■ Distribution through pip
■ Plugins remain independently usable/testable
■ Only user-facing if your users are developers
Other systems and use cases
□ Mercurial extensions
□ pytest plugins (pluggy)
□ pylint plugins
□ flake8 plugins
■ Command extensions
□ SQLAlchemy dialects/DBAPIs
□ Sphinx extensions
□ Buildout extensions
□ Dectate and Reg
We learned that plugins are:
□ Redistributable components for composing behavior
□ The architecture for scaling development
■ Internal / External
■ Open / Closed-source
□ Not all plugin implementations are the same.
□ Still plenty of room for innovation!