Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

Transcript

1. Tyler Langlois, October 12th 2017
   Software Engineer, Elastic
   @leothrix, github: tylerjl
   Custom Types and
   Providers: Modeling Modern
   REST Interfaces and Beyond
   

   View Slide

2. 2
   Obligatory “About Me” Slide
   • Been with company since 2014
   • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and
   Kibana)
   • Puppet-ing in one way or another over my whole professional career
   • Brought too many Elastic stickers that need to be given away (please
   partake)
   • Talk to me about Elasticsearch/Logstash/Kibana/Beats!
   Infrastructure/Operations/Software Engineer @ Elastic
   

   View Slide

3. 3
   Who is This Presentation For?
   Developers who work with Puppet modules
   Puppet users who want to dip into native type/provider development
   “What in the %@#$ is the Elasticsearch module doing”
   Operators who want to automate against APIs
   Hopefully empowers you to implement custom resources on your own
   

   View Slide

4. 4
   WHAT DOES THIS TALK’S
   TITLE EVEN MEAN?
   

   View Slide

5. View Slide

6. 6
   Managing Resources
   with Raw APIs
   Example: CloudFormation
   • Pro:
   • Infrastructure resources are data
   • Extensible
   • Con:
   • Managing changes
   • Grokking huge chunks of JSON
   

   View Slide

7. 7
   Modeling Resources with
   Raw APIs
   Example: Terraform
   • Pro:
   • Readable
   • Manageable
   • Lifecycle + changes
   • Interoperability between other
   systems
   

   View Slide

8. 8
   Modeling Resources in Puppet
   A DSL to Model
   Disparate Resources
   A Graph to Manage
   Relationships
   A Concept of
   Changes to
   Manage Lifecycles
   ls, stat,
   chmod,
   chown
   sysv,
   systemd,
   upstart
   deb, rpm,
   pkg
   }
   

   View Slide

9. 9
   Modeling Resources in Puppet
   Abstraction is Powerful
   file { “/tmp/foo”:
   source => “puppet:///foo”,
   } ->
   package { “foo”:
   source => “/tmp/foo”
   } ~>
   service { “foo”:
   ensure => “running”,
   }
   

   View Slide

10. 10
    Modeling Resources in Puppet
    ?
    Elasticsearch
    Logstash
    Other
    REST
    APIs
    }
    

    View Slide

11. 11
    Modeling Resources in Puppet
    Extending the idea to APIs
    elasticsearch::template { “logstash”:
    content => {
    “template” => “*”,
    “settings” => {
    “number_of_replicas” => 0
    }
    }
    } ->
    service { “es-app”:
    ensure => “running”
    }
    

    View Slide

12. 12
    Modeling Resources in Puppet
    • State Changes
    • Instead of comparing changes with GET responses and template files,
    compare during a no-op
    • A change in state can form dependencies and refresh events into other
    resources
    • Trickling changes up via reports lends better visibility
    Benefits
    

    View Slide

13. 13
    Modeling Resources in Puppet
    • State Changes
    • More fine-grained control
    • Most resources can be represented as Puppet hashes, so Hiera can
    be fully leveraged
    • Communicating via full Ruby HTTP libraries means CA files, auth, and
    more are easier to control
    • TESTS!
    Benefits
    

    View Slide

14. 14
    Modeling Resources in Puppet
    • State Changes
    • More fine-grained control
    • Some existing API-based resources:
    • Kubernetes module (swagger-generated)
    • Google Cloud
    • Following examples will be low-level (i.e. with just native Ruby HTTP
    libraries)
    • …hopefully, will help you write your own for $system
    

    View Slide

15. 15
    Let’s (briefly) talk
    about Puppet Types
    and Providers
    

    View Slide

16. 16
    Types, Providers, and their Resources
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type
    • Has some way to change a
    property
    • Its state is introspectable and
    discoverable
    • Uniquely identified
    • How Ruby interacts with
    actual commands/system
    properties
    • Knows how to discover the
    properties of resources
    • Normalized provider API to
    Puppet DSL
    • Somewhat typed, catalog
    compilation
    • Abstraction over providers
    

    View Slide

17. 17
    Types, Providers, and their Resources: service
    • systemctl/service/rc
    commands
    • Startup visibility with enable/
    chkconfig/etc.
    • Primarily shell-based for state
    • One provider for each init
    system
    • Ruby knows which shell
    commands to invoke to start,
    stop, enable, etc.
    • Unified API to start, enable,
    and restart a general
    service resource
    • Abstraction over provider-
    specific implementations
    • What we see in a manifest
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type
    

    View Slide

18. 18
    Types, Providers, and their Resources: elasticsearch
    • REST API endpoints
    • Objects modeled in JSON
    • Individual endpoints via
    _template, _ingest, etc.
    • One provider base class, one
    provider per resource type
    • Using native Ruby HTTP APIs
    are high-level enough
    • Better alternative than `exec
    { “curl”:`
    • Resource properties
    expressed in Puppet DSL
    hashes
    • We don’t make API calls, we
    declare desired state
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type
    

    View Slide

19. 19
    Then:
    

    View Slide

20. 20
    Now:
    

    View Slide

21. Case Study:
    Elasticsearch
    Pipelines
    curl vs. Puppet
    

    View Slide

22. 22
    Ingest Pipelines
    

    View Slide

23. 23
    Ingest Pipelines
    

    View Slide

24. 24
    Ingest Pipelines
    • All pipelines are uniquely identified by a name (like defined or native types!)
    • Endpoints to manage pipelines:
    • GET to retrieve JSON object enumerating all pipelines
    • Note: can also retrieved based by name alone
    • PUT to create with JSON body
    • Note that we’re using unauthenticated APIs right now
    Key observations
    

    View Slide

25. 25
    Ingest Pipelines: Puppet Type
    

    View Slide

26. 26
    Ingest Pipelines: Puppet Type (Implementation)
    

    View Slide

27. 27
    Ingest Pipelines: Puppet Type (Implementation)
    …what the included abstraction does
    

    View Slide

28. 28
    Ingest Pipelines: Puppet Provider (Implementation)
    

    View Slide

29. 29
    Ingest Pipelines: Puppet Provider (details)
    …what the parent class does
    

    View Slide

30. 30
    Ingest Pipelines: Puppet Provider (details)
    …what the parent class does
    

    View Slide

31. 31
    Ingest Pipelines: Puppet Tests
    

    View Slide

32. 32
    Ingest Pipelines
    • That’s most of it!
    • Test-driven development + rspec makes it smooth
    • Bulk is abstracted; the beefy parts are in parent classes and reused by
    templates, indices, etc.
    • Native types and providers ≠ scary
    Summary
    

    View Slide

33. 33
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4
    

    View Slide

34. View Slide

35. 35
    An Example of Returning a Hash to Prefetch
    Automatically Gathering Resources
    uri = URI(“http://localhost:9200/_template”)
    http = Net::HTTP.new uri.host, uri.port
    req = Net::HTTP::Get.new uri.request_uri
    response = http.request req
    JSON.parse(response.body).map do |object_name, api_object|
    {
    :name => object_name,
    :ensure => :present,
    :content => api_object,
    :provider => name
    }
    end
    

    View Slide

36. 36
    Advantages
    • puppet resource functionality
    • Minimizes chatter with API endpoints
    • i.e., checking for existence versus properties, etc.
    • Call flush only when necessary
    • Additional API freebies (i.e., centralized access in flush(), etc.)
    Prefetching resources versus vanilla exists?
    

    View Slide

37. View Slide

38. 38
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4
    

    View Slide

39. 39
    Response Content vs. Request Content
    Usually never 1:1 mappings
    {
    "logstash": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
    "logstash-*"
    ],
    . . .
    elasticsearch::template {
    'logstash':
    content => {
    'template' =>
    ‘logstash-*’,
    'settings' => {
    . . .
    vs.
    

    View Slide

40. 40
    Types To the Rescue
    • A resource’s desired state is almost never the plain response for a query
    against the resource
    • Example: kubernetes Deployment versus the state of a Deployment
    • munge can help unify the resource versus JSON for comparability
    • insync? can be enhanced to understand which fields are being explicitly
    controlled by a user
    • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in
    {“another”: “field”}
    • Used pretty heavily in puppet-elasticsearch
    Managing response data
    

    View Slide

41. 41
    Example: Setting Default Fields
    Elasticsearch template
    # Set default values for templates
    munge do |value|
    {
    'order' => 0,
    'aliases' => {},
    'mappings' => {}
    }.merge(value)
    end
    

    View Slide

42. 42
    Example: Unifying Formatting
    Elasticsearch template
    # Normalize then compare the Puppet hash and json
    def insync?(is)
    Puppet_X::Elastic.deep_implode(is) == \
    Puppet_X::Elastic.deep_implode(should)
    end
    { “foo”:
    { “bar”: “value” }
    }
    {
    “foo.bar”: “value”
    }
    

    View Slide

43. 43
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4
    

    View Slide

44. 44
    HTTP In Providers
    

    View Slide

45. 45
    HTTP In Providers
    • Native HTTP libraries let us more easily control and pass:
    • TLS certificate authorities and verification booleans
    • HTTP basic auth credentials
    • Failure cases (timeouts, 4xx/5xx response codes, etc.)
    • In this case with Elasticsearch, error responses can return JSON
    messages for more helpful Puppet failures
    

    View Slide

46. 46
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4
    

    View Slide

47. 47
    API Availability
    • What happens if:
    • An API-based REST resource requires an API to be up, not just a
    daemon?
    • A resource should block until one is available?
    • An unrelated resource needs that API as well?
    Weird edge cases when controlling APIs as opposed to hosts
    

    View Slide

48. 48
    API Availability
    • es_instance_conn_validator doesn’t resolve until a connection can
    be made
    

    View Slide

49. Some observations
    after a couple years…
    

    View Slide

50. 50
    Results From the Field
    • One parent class makes
    creating more easy
    • Supported REST-based
    resources include:
    • indices
    • templates
    • pipelines
    • + more
    Extensibility
    • rspec + webmock for
    great testing
    • ES docs + specs first
    have made some
    implementations first try
    successes
    • Good mocks make some
    acceptance tests
    unnecessary (faster CI!)
    Reliability
    • Much easier to extend to
    new OS’s (i.e., Windows)
    • Greater control has made
    some tasks (like 3.x →
    4.x module update)
    smooth
    + more
    

    View Slide

51. 51
    Questions?
    • github.com/elastic
    Thank You!
    

    View Slide