A Plug for Plugins (PyGotham 2017)

The philosophy of small, focused software has its merits, but for most of us, software is big. From your browser to your kernel, the size of these applications is lost on most of us. How do these programs even grow to this size?

In all of these cases, the answer involves an often-overlooked pattern: the plugin. Call them modules or extensions, if you’d prefer, but they are wildly successful. In fact, the only thing wider than the success of plugin-based architecture is the variety of implementations, especially in Python’s dynamic environment.

This talk covers the basis for plugins, reviews Python’s current offerings using examples, and provides guidance as to the bright future of plugin architecture.

Transcript

1. A
   Plug for
   Plugins
   PyGotham 2017
   Mahmoud Hashemi
   

   View Slide

2. Plugins
   Add-ons, components,
   modules, themes, widgets,
   extensions
   

   View Slide

3. Plugins
   Pieces of software,
   Loaded by a core program,
   To add behaviors.
   (a haiku)
   

   View Slide

4. Brand issues
   with plugins
   Functional, but not very dignified.
   

   View Slide

5. Brand issues
   with plugins
   Tired, possibly a code smell?
   

   View Slide

6. Brand issues
   with plugins
   Or maybe why WordPress breaks so much?
   Tired, possibly a code smell?
   

   View Slide

7. Brand issues
   with plugins
   Eclipsed
   by some
   very bad
   experiences.
   

   View Slide

8. Plugins everywhere!
   □ Browser extensions
   □ Kernel modules
   □ Shells (.bashrc & .bash_profile)
   □ Good editors (from Sublime to emacs)
   □ So many more!
   When plugins work, you might not even notice.
   

   View Slide

9. Plugins!
   Up next:
   □ Why plugins exist
   □ Reasons to use plugins
   □ Using plugins in Python applications
   

   View Slide

10. Stages of development skills
    □ Sticking values together
    >>> 2 + 2
    □ Sticking lines together
    □ Sticking functions together
    □ Sticking modules together
    □ Sticking behaviors together
    (Unsticking those modules & functions)
    

    View Slide

11. When to use plugin architecture
    □ Reduced core size
    □ Work around licensing issues
    □ Community development
    ■ Establish an API for code integration
    ■ Especially for applications
    

    View Slide

12. Case study: conda cli
    Simple plugins, used internally
    

    View Slide

13. conda
    The Python-first, cross-platform package manager.
    (Imagine if pip and virtualenv had more-powerful fusion form)
    

    View Slide

14. conda internals
    Inside of https://github.com/conda/conda:
    

    View Slide

15. conda install plugin
    cli/main_install.py:
    

    View Slide

16. conda core
    cli/main.py:
    

    View Slide

17. Pros and cons of conda’s plugins
    □ Pros
    ■ Simple
    ■ Dynamic loading of hooks
    ■ Only requires standard library
    □ Cons
    ■ Not systemized (copyable but not general)
    ■ Not user-facing
    

    View Slide

18. Case study: pluginbase
    Popular reusable plugin system
    

    View Slide

19. pluginbase
    From the makers of Flask and Click, 3-step plugins:
    https://github.com/pluginbase/pluginbase
    

    View Slide

20. Pros and cons of pluginbase
    □ Pros
    ■ Generalized
    ■ User-facing
    □ Cons
    ■ No distribution story
    ■ Path wrangling
    ■ Attempts to:
    ➢ “Localize” modules
    ➢ Enable simultaneous loading of
    multiple versions of plugins
    

    View Slide

21. Let’s talk about modules
    □ Python’s built-in plugin system
    □ Simple, but also not so much
    ■ PEP 302 Finders and Loaders
    □ Global by design
    ■ sys.modules
    

    View Slide

22. Case study: gather
    Shiny new entrypoints-based system
    

    View Slide

23. First: Entry Points
    One of many setuptools features:
    “Entry points are a simple way for distributions to
    ‘advertise’ Python functions for use by other packages.”
    For instance:
    import pkg_resources
    for entry_point in pkg_resources.iter_entry_points('label'):
    registry[entry_point.name] = entry_point.load()
    http://amir.rachum.com/blog/2017/07/28/python-entry-points/
    

    View Slide

24. gather
    First, we need an application.
    core_app.py:
    

    View Slide

25. gather
    Now we need to build a plugin.
    core_app_plugin.py:
    (We celebrate usefulness.)
    

    View Slide

26. gather
    And now the gather entry point special sauce:
    

    View Slide

27. gather
    And once we pip install everything, it’s time to celebrate:
    

    View Slide

28. Pros and cons of gather
    □ Pros
    ■ Uses standard setuptools
    ■ Developer-facing
    ■ Distribution through pip
    ■ Plugins remain independently usable/testable
    □ Cons
    ■ Only user-facing if your users are developers
    

    View Slide

29. Other systems and use cases
    □ stevedore
    □ twisted.plugin
    □ Mercurial extensions
    □ pytest plugins (pluggy)
    □ venusian
    □ straight.plugin
    □ pylint plugins
    □ flake8 plugins
    □ zope.component
    □ Django
    ■ Apps
    ■ Middlewares
    ■ Command extensions
    □ SQLAlchemy dialects/DBAPIs
    □ Sphinx extensions
    □ Buildout extensions
    □ Pike
    □ Dectate and Reg
    

    View Slide

30. Wrapping up
    We learned that plugins are:
    □ Redistributable components for composing behavior
    □ The architecture for scaling development
    ■ Internal / External
    ■ Open / Closed-source
    □ Not all plugin implementations are the same.
    □ Still plenty of room for innovation!
    

    View Slide

31. Questions?
    @mhashemi
    github.com/mahmoud
    sedimental.org/plugin_systems.html
    ?
    

    View Slide