A Plug for Plugins (PyGotham 2017)

A Plug for Plugins (PyGotham 2017)

The philosophy of small, focused software has its merits, but for most of us, software is big. From your browser to your kernel, the size of these applications is lost on most of us. How do these programs even grow to this size?

In all of these cases, the answer involves an often-overlooked pattern: the plugin. Call them modules or extensions, if you’d prefer, but they are wildly successful. In fact, the only thing wider than the success of plugin-based architecture is the variety of implementations, especially in Python’s dynamic environment.

This talk covers the basis for plugins, reviews Python’s current offerings using examples, and provides guidance as to the bright future of plugin architecture.


Mahmoud Hashemi

October 08, 2017


  1. A Plug for Plugins PyGotham 2017 Mahmoud Hashemi

  2. Plugins Add-ons, components, modules, themes, widgets, extensions

  3. Plugins Pieces of software, Loaded by a core program, To

    add behaviors. (a haiku)
  4. Brand issues with plugins Functional, but not very dignified.

  5. Brand issues with plugins Tired, possibly a code smell?

  6. Brand issues with plugins Or maybe why WordPress breaks so

    much? Tired, possibly a code smell?
  7. Brand issues with plugins Eclipsed by some very bad experiences.

  8. Plugins everywhere! □ Browser extensions □ Kernel modules □ Shells

    (.bashrc & .bash_profile) □ Good editors (from Sublime to emacs) □ So many more! When plugins work, you might not even notice.
  9. Plugins! Up next: □ Why plugins exist □ Reasons to

    use plugins □ Using plugins in Python applications
  10. Stages of development skills □ Sticking values together >>> 2

    + 2 □ Sticking lines together □ Sticking functions together □ Sticking modules together □ Sticking behaviors together (Unsticking those modules & functions)
  11. When to use plugin architecture □ Reduced core size □

    Work around licensing issues □ Community development ▪ Establish an API for code integration ▪ Especially for applications
  12. Case study: conda cli Simple plugins, used internally

  13. conda The Python-first, cross-platform package manager. (Imagine if pip and

    virtualenv had more-powerful fusion form)
  14. conda internals Inside of https://github.com/conda/conda:

  15. conda install plugin cli/main_install.py:

  16. conda core cli/main.py:

  17. Pros and cons of conda’s plugins □ Pros ▪ Simple

    ▪ Dynamic loading of hooks ▪ Only requires standard library □ Cons ▪ Not systemized (copyable but not general) ▪ Not user-facing
  18. Case study: pluginbase Popular reusable plugin system

  19. pluginbase From the makers of Flask and Click, 3-step plugins:

  20. Pros and cons of pluginbase □ Pros ▪ Generalized ▪

    User-facing □ Cons ▪ No distribution story ▪ Path wrangling ▪ Attempts to: ➢ “Localize” modules ➢ Enable simultaneous loading of multiple versions of plugins
  21. Let’s talk about modules □ Python’s built-in plugin system □

    Simple, but also not so much ▪ PEP 302 Finders and Loaders □ Global by design ▪ sys.modules
  22. Case study: gather Shiny new entrypoints-based system

  23. First: Entry Points One of many setuptools features: “Entry points

    are a simple way for distributions to ‘advertise’ Python functions for use by other packages.” For instance: import pkg_resources for entry_point in pkg_resources.iter_entry_points('label'): registry[entry_point.name] = entry_point.load() http://amir.rachum.com/blog/2017/07/28/python-entry-points/
  24. gather First, we need an application. core_app.py:

  25. gather Now we need to build a plugin. core_app_plugin.py: (We

    celebrate usefulness.)
  26. gather And now the gather entry point special sauce:

  27. gather And once we pip install everything, it’s time to

  28. Pros and cons of gather □ Pros ▪ Uses standard

    setuptools ▪ Developer-facing ▪ Distribution through pip ▪ Plugins remain independently usable/testable □ Cons ▪ Only user-facing if your users are developers
  29. Other systems and use cases □ stevedore □ twisted.plugin □

    Mercurial extensions □ pytest plugins (pluggy) □ venusian □ straight.plugin □ pylint plugins □ flake8 plugins □ zope.component □ Django ▪ Apps ▪ Middlewares ▪ Command extensions □ SQLAlchemy dialects/DBAPIs □ Sphinx extensions □ Buildout extensions □ Pike □ Dectate and Reg
  30. Wrapping up We learned that plugins are: □ Redistributable components

    for composing behavior □ The architecture for scaling development ▪ Internal / External ▪ Open / Closed-source □ Not all plugin implementations are the same. □ Still plenty of room for innovation!
  31. Questions? @mhashemi github.com/mahmoud sedimental.org/plugin_systems.html ?