Dynamic Code Patterns: Extending Your Applications with Plugins by Doug Hellmann

Python makes loading code dynamically easy, allowing you to configure and extend your application by discovering and loading extensions at runtime. This presentation will discuss the techniques for dynamic code loading used in several well-known applications and weigh the pros and cons of each approach.

PyCon 2013

Transcript

1. Dynamic Code Patterns:
   Extending Your
   Applications with Plugins
   Doug Hellmann – @doughellmann – PyCon 2013
   1
   Saturday, March 16, 13
   

   View Slide

2. What is a “Plugin”?
   • Loaded Dynamically
   • Extends Core
   • Possibly Unknown Source
   2
   Saturday, March 16, 13
   

   View Slide

3. Why Plugins?
   • Better API Abstraction
   • Reduce Core Dependencies
   • Strategy Pattern
   • Visitor Pattern
   • Indirect Code Contributions
   3
   Saturday, March 16, 13
   

   View Slide

4. • OpenStack Metering
   • Measures Clouds
   • Varied Billing Requirements
   • Deployers Extend and Customize
   Ceilometer
   4
   Saturday, March 16, 13
   

   View Slide

5. Ceilometer Design
   Compute
   Images
   Volumes
   Network
   Objects
   Notification
   Message Bus
   Event
   Listener
   Compute
   Pollsters
   Central
   Pollsters
   Collector
   Meter Data
   Message Bus
   5
   Saturday, March 16, 13
   

   View Slide

6. Ceilometer Design
   Compute
   Images
   Volumes
   Network
   Objects
   Notification
   Message Bus
   Event
   Listener
   Compute
   Pollsters
   Central
   Pollsters
   Collector
   Meter Data
   Message Bus
   5
   Saturday, March 16, 13
   

   View Slide

7. Ceilometer Design
   Compute
   Images
   Volumes
   Network
   Objects
   Notification
   Message Bus
   Event
   Listener
   Compute
   Pollsters
   Central
   Pollsters
   Collector
   Meter Data
   Message Bus
   5
   Saturday, March 16, 13
   

   View Slide

8. Ceilometer Design
   Compute
   Images
   Volumes
   Network
   Objects
   Notification
   Message Bus
   Event
   Listener
   Compute
   Pollsters
   Central
   Pollsters
   Collector
   Meter Data
   Message Bus
   5
   Saturday, March 16, 13
   

   View Slide

9. Ceilometer Design
   Compute
   Images
   Volumes
   Network
   Objects
   Notification
   Message Bus
   Event
   Listener
   Compute
   Pollsters
   Central
   Pollsters
   Collector
   Meter Data
   Message Bus
   5
   Saturday, March 16, 13
   

   View Slide

10. Ceilometer Design
    Compute
    Images
    Volumes
    Network
    Objects
    Notification
    Message Bus
    Event
    Listener
    Compute
    Pollsters
    Central
    Pollsters
    Collector
    Meter Data
    Message Bus
    5
    Saturday, March 16, 13
    

    View Slide

11. Ceilometer Design
    Compute
    Images
    Volumes
    Network
    Objects
    Notification
    Message Bus
    Event
    Listener
    Compute
    Pollsters
    Central
    Pollsters
    Collector
    Meter Data
    Message Bus
    5
    Saturday, March 16, 13
    

    View Slide

12. Ceilometer Design
    Compute
    Images
    Volumes
    Network
    Objects
    Notification
    Message Bus
    Event
    Listener
    Compute
    Pollsters
    Central
    Pollsters
    Collector
    Meter Data
    Message Bus
    5
    Saturday, March 16, 13
    

    View Slide

13. Ceilometer Design
    Compute
    Images
    Volumes
    Network
    Objects
    Notification
    Message Bus
    Event
    Listener
    Compute
    Pollsters
    Central
    Pollsters
    Collector
    Meter Data
    Message Bus
    5
    Saturday, March 16, 13
    

    View Slide

14. Ceilometer Plugins
    • Message Bus
    • Receiving Notifications
    • Polling Compute
    • Polling Other
    • Storage
    6
    Saturday, March 16, 13
    

    View Slide

15. Research
    • Blogofile
    • Sphinx
    • Mercurial
    • cliff
    • virtualenvwrapper
    • Nose
    • Trac
    • Django
    • Pyramid
    • SQLAlchemy
    • Diamond
    • Nova
    7
    Saturday, March 16, 13
    

    View Slide

16. Discovery
    Explicit Scan
    File
    Import
    Reference
    Mercurial
    Diamond
    Blogofile
    Mercurial
    Django
    Pyramid
    Sphinx
    Nova
    Trac
    Nose
    cliff
    virtualenvwrapper
    8
    Saturday, March 16, 13
    

    View Slide

17. Enabling
    Explicit Implicit
    Django
    Pyramid
    SQLAlchemy
    Blogofile
    Mercurial
    Trac
    Sphinx
    virtualenvwrapper
    cliff
    9
    Saturday, March 16, 13
    

    View Slide

18. Importing
    Custom pkg_resources
    Django
    Pyramid
    Sphinx
    Diamond
    Nova
    Nose
    SQLAlchemy
    Blogofile
    Trac
    Nose
    SQLAlchemy
    Blogofile
    cliff
    virtualenvwrapper
    10
    Saturday, March 16, 13
    

    View Slide

19. Integration
    Fine Coarse
    Prompt
    Inspect
    Nose
    virtualenvwrapper
    Sphinx
    Trac
    cliff
    Diamond
    Mercurial
    Blogofile
    Pyramid
    Django
    11
    Saturday, March 16, 13
    

    View Slide

20. Integration
    Fine Coarse
    Prompt
    Inspect
    Nose
    virtualenvwrapper
    Sphinx
    Trac
    cliff
    Diamond
    Mercurial
    Blogofile
    Pyramid
    Django
    12
    Saturday, March 16, 13
    

    View Slide

21. API Enforcement
    Convention Base Class / Interface
    Blogofile
    Mercurial
    Django
    Sphinx
    Pyramid
    virtualenvwrapper
    Nose (optional)
    Trac (interface)
    Diamond
    cliff (abc)
    13
    Saturday, March 16, 13
    

    View Slide

22. Invocation
    Driver Dispatcher Iterator
    SQLAlchemy
    Nova
    Mercurial
    Django
    Pyramid
    cliff
    Nose
    Diamond
    virtualenvwrapper
    14
    Saturday, March 16, 13
    

    View Slide

23. Discovery / Importing
    • Entry Points
    • Distribute and pkg_resources
    • Be Consistent
    15
    Saturday, March 16, 13
    

    View Slide

24. Enabling
    • Explicit disabling
    • Automatic disabling
    16
    Saturday, March 16, 13
    

    View Slide

25. Integration
    • Fine
    • Inspect
    • Application Owns Relationship
    17
    Saturday, March 16, 13
    

    View Slide

26. API Enforcement
    • Abstract Base Classes
    • Duck Typing
    18
    Saturday, March 16, 13
    

    View Slide

27. Invocation
    • Storage – Driver
    • Notifications – Dispatcher
    • Pollsters – Iterator
    19
    Saturday, March 16, 13
    

    View Slide

28. stevedore
    • Implements Patterns
    • Wraps pkg_resources
    20
    Saturday, March 16, 13
    

    View Slide

29. NamedExtensionManager
    • Multiple Plugins
    • Only Loads Named Plugins
    • map()
    21
    Saturday, March 16, 13
    

    View Slide

30. EnabledExtensionManager
    • Multiple Plugins
    • Checks Each with Function on Load
    • map()
    22
    Saturday, March 16, 13
    

    View Slide

31. DispatchExtensionManager
    • Multiple Plugins
    • Invokes Subset on map()
    23
    Saturday, March 16, 13
    

    View Slide

32. DriverManager
    • Single Plugin
    • Direct Access
    24
    Saturday, March 16, 13
    

    View Slide

33. Dynamic Code Patterns:
    Extending Your
    Applications with Plugins
    Doug Hellmann – @doughellmann – PyCon 2013
    http://packages.python.org/distribute/pkg_resources.html
    https://github.com/dreamhost/stevedore
    https://launchpad.net/ceilometer
    http://doughellmann.com
    25
    Saturday, March 16, 13
    

    View Slide