Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Plug for Plugins (PyGotham 2017)

A Plug for Plugins (PyGotham 2017)

The philosophy of small, focused software has its merits, but for most of us, software is big. From your browser to your kernel, the size of these applications is lost on most of us. How do these programs even grow to this size?

In all of these cases, the answer involves an often-overlooked pattern: the plugin. Call them modules or extensions, if you’d prefer, but they are wildly successful. In fact, the only thing wider than the success of plugin-based architecture is the variety of implementations, especially in Python’s dynamic environment.

This talk covers the basis for plugins, reviews Python’s current offerings using examples, and provides guidance as to the bright future of plugin architecture.

Mahmoud Hashemi

October 08, 2017
Tweet

More Decks by Mahmoud Hashemi

Other Decks in Programming

Transcript

  1. A
    Plug for
    Plugins
    PyGotham 2017
    Mahmoud Hashemi

    View Slide

  2. Plugins
    Add-ons, components,
    modules, themes, widgets,
    extensions

    View Slide

  3. Plugins
    Pieces of software,
    Loaded by a core program,
    To add behaviors.
    (a haiku)

    View Slide

  4. Brand issues
    with plugins
    Functional, but not very dignified.

    View Slide

  5. Brand issues
    with plugins
    Tired, possibly a code smell?

    View Slide

  6. Brand issues
    with plugins
    Or maybe why WordPress breaks so much?
    Tired, possibly a code smell?

    View Slide

  7. Brand issues
    with plugins
    Eclipsed
    by some
    very bad
    experiences.

    View Slide

  8. Plugins everywhere!
    □ Browser extensions
    □ Kernel modules
    □ Shells (.bashrc & .bash_profile)
    □ Good editors (from Sublime to emacs)
    □ So many more!
    When plugins work, you might not even notice.

    View Slide

  9. Plugins!
    Up next:
    □ Why plugins exist
    □ Reasons to use plugins
    □ Using plugins in Python applications

    View Slide

  10. Stages of development skills
    □ Sticking values together
    >>> 2 + 2
    □ Sticking lines together
    □ Sticking functions together
    □ Sticking modules together
    □ Sticking behaviors together
    (Unsticking those modules & functions)

    View Slide

  11. When to use plugin architecture
    □ Reduced core size
    □ Work around licensing issues
    □ Community development
    ■ Establish an API for code integration
    ■ Especially for applications

    View Slide

  12. Case study: conda cli
    Simple plugins, used internally

    View Slide

  13. conda
    The Python-first, cross-platform package manager.
    (Imagine if pip and virtualenv had more-powerful fusion form)

    View Slide

  14. conda internals
    Inside of https://github.com/conda/conda:

    View Slide

  15. conda install plugin
    cli/main_install.py:

    View Slide

  16. conda core
    cli/main.py:

    View Slide

  17. Pros and cons of conda’s plugins
    □ Pros
    ■ Simple
    ■ Dynamic loading of hooks
    ■ Only requires standard library
    □ Cons
    ■ Not systemized (copyable but not general)
    ■ Not user-facing

    View Slide

  18. Case study: pluginbase
    Popular reusable plugin system

    View Slide

  19. pluginbase
    From the makers of Flask and Click, 3-step plugins:
    https://github.com/pluginbase/pluginbase

    View Slide

  20. Pros and cons of pluginbase
    □ Pros
    ■ Generalized
    ■ User-facing
    □ Cons
    ■ No distribution story
    ■ Path wrangling
    ■ Attempts to:
    ➢ “Localize” modules
    ➢ Enable simultaneous loading of
    multiple versions of plugins

    View Slide

  21. Let’s talk about modules
    □ Python’s built-in plugin system
    □ Simple, but also not so much
    ■ PEP 302 Finders and Loaders
    □ Global by design
    ■ sys.modules

    View Slide

  22. Case study: gather
    Shiny new entrypoints-based system

    View Slide

  23. First: Entry Points
    One of many setuptools features:
    “Entry points are a simple way for distributions to
    ‘advertise’ Python functions for use by other packages.”
    For instance:
    import pkg_resources
    for entry_point in pkg_resources.iter_entry_points('label'):
    registry[entry_point.name] = entry_point.load()
    http://amir.rachum.com/blog/2017/07/28/python-entry-points/

    View Slide

  24. gather
    First, we need an application.
    core_app.py:

    View Slide

  25. gather
    Now we need to build a plugin.
    core_app_plugin.py:
    (We celebrate usefulness.)

    View Slide

  26. gather
    And now the gather entry point special sauce:

    View Slide

  27. gather
    And once we pip install everything, it’s time to celebrate:

    View Slide

  28. Pros and cons of gather
    □ Pros
    ■ Uses standard setuptools
    ■ Developer-facing
    ■ Distribution through pip
    ■ Plugins remain independently usable/testable
    □ Cons
    ■ Only user-facing if your users are developers

    View Slide

  29. Other systems and use cases
    □ stevedore
    □ twisted.plugin
    □ Mercurial extensions
    □ pytest plugins (pluggy)
    □ venusian
    □ straight.plugin
    □ pylint plugins
    □ flake8 plugins
    □ zope.component
    □ Django
    ■ Apps
    ■ Middlewares
    ■ Command extensions
    □ SQLAlchemy dialects/DBAPIs
    □ Sphinx extensions
    □ Buildout extensions
    □ Pike
    □ Dectate and Reg

    View Slide

  30. Wrapping up
    We learned that plugins are:
    □ Redistributable components for composing behavior
    □ The architecture for scaling development
    ■ Internal / External
    ■ Open / Closed-source
    □ Not all plugin implementations are the same.
    □ Still plenty of room for innovation!

    View Slide

  31. Questions?
    @mhashemi
    github.com/mahmoud
    sedimental.org/plugin_systems.html
    ?

    View Slide