Slide 1

Slide 1 text

A Plug for Plugins PyGotham 2017 Mahmoud Hashemi

Slide 2

Slide 2 text

Plugins Add-ons, components, modules, themes, widgets, extensions

Slide 3

Slide 3 text

Plugins Pieces of software, Loaded by a core program, To add behaviors. (a haiku)

Slide 4

Slide 4 text

Brand issues with plugins Functional, but not very dignified.

Slide 5

Slide 5 text

Brand issues with plugins Tired, possibly a code smell?

Slide 6

Slide 6 text

Brand issues with plugins Or maybe why WordPress breaks so much? Tired, possibly a code smell?

Slide 7

Slide 7 text

Brand issues with plugins Eclipsed by some very bad experiences.

Slide 8

Slide 8 text

Plugins everywhere! □ Browser extensions □ Kernel modules □ Shells (.bashrc & .bash_profile) □ Good editors (from Sublime to emacs) □ So many more! When plugins work, you might not even notice.

Slide 9

Slide 9 text

Plugins! Up next: □ Why plugins exist □ Reasons to use plugins □ Using plugins in Python applications

Slide 10

Slide 10 text

Stages of development skills □ Sticking values together >>> 2 + 2 □ Sticking lines together □ Sticking functions together □ Sticking modules together □ Sticking behaviors together (Unsticking those modules & functions)

Slide 11

Slide 11 text

When to use plugin architecture □ Reduced core size □ Work around licensing issues □ Community development ■ Establish an API for code integration ■ Especially for applications

Slide 12

Slide 12 text

Case study: conda cli Simple plugins, used internally

Slide 13

Slide 13 text

conda The Python-first, cross-platform package manager. (Imagine if pip and virtualenv had more-powerful fusion form)

Slide 14

Slide 14 text

conda internals Inside of https://github.com/conda/conda:

Slide 15

Slide 15 text

conda install plugin cli/main_install.py:

Slide 16

Slide 16 text

conda core cli/main.py:

Slide 17

Slide 17 text

Pros and cons of conda’s plugins □ Pros ■ Simple ■ Dynamic loading of hooks ■ Only requires standard library □ Cons ■ Not systemized (copyable but not general) ■ Not user-facing

Slide 18

Slide 18 text

Case study: pluginbase Popular reusable plugin system

Slide 19

Slide 19 text

pluginbase From the makers of Flask and Click, 3-step plugins: https://github.com/pluginbase/pluginbase

Slide 20

Slide 20 text

Pros and cons of pluginbase □ Pros ■ Generalized ■ User-facing □ Cons ■ No distribution story ■ Path wrangling ■ Attempts to: ➢ “Localize” modules ➢ Enable simultaneous loading of multiple versions of plugins

Slide 21

Slide 21 text

Let’s talk about modules □ Python’s built-in plugin system □ Simple, but also not so much ■ PEP 302 Finders and Loaders □ Global by design ■ sys.modules

Slide 22

Slide 22 text

Case study: gather Shiny new entrypoints-based system

Slide 23

Slide 23 text

First: Entry Points One of many setuptools features: “Entry points are a simple way for distributions to ‘advertise’ Python functions for use by other packages.” For instance: import pkg_resources for entry_point in pkg_resources.iter_entry_points('label'): registry[entry_point.name] = entry_point.load() http://amir.rachum.com/blog/2017/07/28/python-entry-points/

Slide 24

Slide 24 text

gather First, we need an application. core_app.py:

Slide 25

Slide 25 text

gather Now we need to build a plugin. core_app_plugin.py: (We celebrate usefulness.)

Slide 26

Slide 26 text

gather And now the gather entry point special sauce:

Slide 27

Slide 27 text

gather And once we pip install everything, it’s time to celebrate:

Slide 28

Slide 28 text

Pros and cons of gather □ Pros ■ Uses standard setuptools ■ Developer-facing ■ Distribution through pip ■ Plugins remain independently usable/testable □ Cons ■ Only user-facing if your users are developers

Slide 29

Slide 29 text

Other systems and use cases □ stevedore □ twisted.plugin □ Mercurial extensions □ pytest plugins (pluggy) □ venusian □ straight.plugin □ pylint plugins □ flake8 plugins □ zope.component □ Django ■ Apps ■ Middlewares ■ Command extensions □ SQLAlchemy dialects/DBAPIs □ Sphinx extensions □ Buildout extensions □ Pike □ Dectate and Reg

Slide 30

Slide 30 text

Wrapping up We learned that plugins are: □ Redistributable components for composing behavior □ The architecture for scaling development ■ Internal / External ■ Open / Closed-source □ Not all plugin implementations are the same. □ Still plenty of room for innovation!

Slide 31

Slide 31 text

Questions? @mhashemi github.com/mahmoud sedimental.org/plugin_systems.html ?