$30 off During Our Annual Pro Sale. View Details »

Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

Tyler L
October 12, 2017

Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

Tyler L

October 12, 2017
Tweet

More Decks by Tyler L

Other Decks in Technology

Transcript

  1. Tyler Langlois, October 12th 2017
    Software Engineer, Elastic
    @leothrix, github: tylerjl
    Custom Types and
    Providers: Modeling Modern
    REST Interfaces and Beyond

    View Slide

  2. 2
    Obligatory “About Me” Slide
    • Been with company since 2014
    • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and
    Kibana)
    • Puppet-ing in one way or another over my whole professional career
    • Brought too many Elastic stickers that need to be given away (please
    partake)
    • Talk to me about Elasticsearch/Logstash/Kibana/Beats!
    Infrastructure/Operations/Software Engineer @ Elastic

    View Slide

  3. 3
    Who is This Presentation For?
    Developers who work with Puppet modules
    Puppet users who want to dip into native type/provider development
    “What in the %@#$ is the Elasticsearch module doing”
    Operators who want to automate against APIs
    Hopefully empowers you to implement custom resources on your own

    View Slide

  4. 4
    WHAT DOES THIS TALK’S
    TITLE EVEN MEAN?

    View Slide

  5. 5

    View Slide

  6. 6
    Managing Resources
    with Raw APIs
    Example: CloudFormation
    • Pro:
    • Infrastructure resources are data
    • Extensible
    • Con:
    • Managing changes
    • Grokking huge chunks of JSON

    View Slide

  7. 7
    Modeling Resources with
    Raw APIs
    Example: Terraform
    • Pro:
    • Readable
    • Manageable
    • Lifecycle + changes
    • Interoperability between other
    systems

    View Slide

  8. 8
    Modeling Resources in Puppet
    A DSL to Model
    Disparate Resources
    A Graph to Manage
    Relationships
    A Concept of
    Changes to
    Manage Lifecycles
    ls, stat,
    chmod,
    chown
    sysv,
    systemd,
    upstart
    deb, rpm,
    pkg
    }

    View Slide

  9. 9
    Modeling Resources in Puppet
    Abstraction is Powerful
    file { “/tmp/foo”:
    source => “puppet:///foo”,
    } ->
    package { “foo”:
    source => “/tmp/foo”
    } ~>
    service { “foo”:
    ensure => “running”,
    }

    View Slide

  10. 10
    Modeling Resources in Puppet
    ?
    Elasticsearch
    Logstash
    Other
    REST
    APIs
    }

    View Slide

  11. 11
    Modeling Resources in Puppet
    Extending the idea to APIs
    elasticsearch::template { “logstash”:
    content => {
    “template” => “*”,
    “settings” => {
    “number_of_replicas” => 0
    }
    }
    } ->
    service { “es-app”:
    ensure => “running”
    }

    View Slide

  12. 12
    Modeling Resources in Puppet
    • State Changes
    • Instead of comparing changes with GET responses and template files,
    compare during a no-op
    • A change in state can form dependencies and refresh events into other
    resources
    • Trickling changes up via reports lends better visibility
    Benefits

    View Slide

  13. 13
    Modeling Resources in Puppet
    • State Changes
    • More fine-grained control
    • Most resources can be represented as Puppet hashes, so Hiera can
    be fully leveraged
    • Communicating via full Ruby HTTP libraries means CA files, auth, and
    more are easier to control
    • TESTS!
    Benefits

    View Slide

  14. 14
    Modeling Resources in Puppet
    • State Changes
    • More fine-grained control
    • Some existing API-based resources:
    • Kubernetes module (swagger-generated)
    • Google Cloud
    • Following examples will be low-level (i.e. with just native Ruby HTTP
    libraries)
    • …hopefully, will help you write your own for $system

    View Slide

  15. 15
    Let’s (briefly) talk
    about Puppet Types
    and Providers

    View Slide

  16. 16
    Types, Providers, and their Resources
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type
    • Has some way to change a
    property
    • Its state is introspectable and
    discoverable
    • Uniquely identified
    • How Ruby interacts with
    actual commands/system
    properties
    • Knows how to discover the
    properties of resources
    • Normalized provider API to
    Puppet DSL
    • Somewhat typed, catalog
    compilation
    • Abstraction over providers

    View Slide

  17. 17
    Types, Providers, and their Resources: service
    • systemctl/service/rc
    commands
    • Startup visibility with enable/
    chkconfig/etc.
    • Primarily shell-based for state
    • One provider for each init
    system
    • Ruby knows which shell
    commands to invoke to start,
    stop, enable, etc.
    • Unified API to start, enable,
    and restart a general
    service resource
    • Abstraction over provider-
    specific implementations
    • What we see in a manifest
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type

    View Slide

  18. 18
    Types, Providers, and their Resources: elasticsearch
    • REST API endpoints
    • Objects modeled in JSON
    • Individual endpoints via
    _template, _ingest, etc.
    • One provider base class, one
    provider per resource type
    • Using native Ruby HTTP APIs
    are high-level enough
    • Better alternative than `exec
    { “curl”:`
    • Resource properties
    expressed in Puppet DSL
    hashes
    • We don’t make API calls, we
    declare desired state
    Underlying
    Resource
    Puppet
    Provider
    Puppet
    Type

    View Slide

  19. 19
    Then:

    View Slide

  20. 20
    Now:

    View Slide

  21. Case Study:
    Elasticsearch
    Pipelines
    curl vs. Puppet

    View Slide

  22. 22
    Ingest Pipelines

    View Slide

  23. 23
    Ingest Pipelines

    View Slide

  24. 24
    Ingest Pipelines
    • All pipelines are uniquely identified by a name (like defined or native types!)
    • Endpoints to manage pipelines:
    • GET to retrieve JSON object enumerating all pipelines
    • Note: can also retrieved based by name alone
    • PUT to create with JSON body
    • Note that we’re using unauthenticated APIs right now
    Key observations

    View Slide

  25. 25
    Ingest Pipelines: Puppet Type

    View Slide

  26. 26
    Ingest Pipelines: Puppet Type (Implementation)

    View Slide

  27. 27
    Ingest Pipelines: Puppet Type (Implementation)
    …what the included abstraction does

    View Slide

  28. 28
    Ingest Pipelines: Puppet Provider (Implementation)

    View Slide

  29. 29
    Ingest Pipelines: Puppet Provider (details)
    …what the parent class does

    View Slide

  30. 30
    Ingest Pipelines: Puppet Provider (details)
    …what the parent class does

    View Slide

  31. 31
    Ingest Pipelines: Puppet Tests

    View Slide

  32. 32
    Ingest Pipelines
    • That’s most of it!
    • Test-driven development + rspec makes it smooth
    • Bulk is abstracted; the beefy parts are in parent classes and reused by
    templates, indices, etc.
    • Native types and providers ≠ scary
    Summary

    View Slide

  33. 33
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4

    View Slide

  34. 34

    View Slide

  35. 35
    An Example of Returning a Hash to Prefetch
    Automatically Gathering Resources
    uri = URI(“http://localhost:9200/_template”)
    http = Net::HTTP.new uri.host, uri.port
    req = Net::HTTP::Get.new uri.request_uri
    response = http.request req
    JSON.parse(response.body).map do |object_name, api_object|
    {
    :name => object_name,
    :ensure => :present,
    :content => api_object,
    :provider => name
    }
    end

    View Slide

  36. 36
    Advantages
    • puppet resource functionality
    • Minimizes chatter with API endpoints
    • i.e., checking for existence versus properties, etc.
    • Call flush only when necessary
    • Additional API freebies (i.e., centralized access in flush(), etc.)
    Prefetching resources versus vanilla exists?

    View Slide

  37. 37

    View Slide

  38. 38
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4

    View Slide

  39. 39
    Response Content vs. Request Content
    Usually never 1:1 mappings
    {
    "logstash": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
    "logstash-*"
    ],
    . . .
    elasticsearch::template {
    'logstash':
    content => {
    'template' =>
    ‘logstash-*’,
    'settings' => {
    . . .
    vs.

    View Slide

  40. 40
    Types To the Rescue
    • A resource’s desired state is almost never the plain response for a query
    against the resource
    • Example: kubernetes Deployment versus the state of a Deployment
    • munge can help unify the resource versus JSON for comparability
    • insync? can be enhanced to understand which fields are being explicitly
    controlled by a user
    • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in
    {“another”: “field”}
    • Used pretty heavily in puppet-elasticsearch
    Managing response data

    View Slide

  41. 41
    Example: Setting Default Fields
    Elasticsearch template
    # Set default values for templates
    munge do |value|
    {
    'order' => 0,
    'aliases' => {},
    'mappings' => {}
    }.merge(value)
    end

    View Slide

  42. 42
    Example: Unifying Formatting
    Elasticsearch template
    # Normalize then compare the Puppet hash and json
    def insync?(is)
    Puppet_X::Elastic.deep_implode(is) == \
    Puppet_X::Elastic.deep_implode(should)
    end
    { “foo”:
    { “bar”: “value” }
    }
    {
    “foo.bar”: “value”
    }

    View Slide

  43. 43
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4

    View Slide

  44. 44
    HTTP In Providers

    View Slide

  45. 45
    HTTP In Providers
    • Native HTTP libraries let us more easily control and pass:
    • TLS certificate authorities and verification booleans
    • HTTP basic auth credentials
    • Failure cases (timeouts, 4xx/5xx response codes, etc.)
    • In this case with Elasticsearch, error responses can return JSON
    messages for more helpful Puppet failures

    View Slide

  46. 46
    Fitting REST Resources Into Puppet
    Considerations
    `exists?` versus `prefetch`
    Leveraging type-level tools
    HTTP
    API availability
    1
    2
    3
    4

    View Slide

  47. 47
    API Availability
    • What happens if:
    • An API-based REST resource requires an API to be up, not just a
    daemon?
    • A resource should block until one is available?
    • An unrelated resource needs that API as well?
    Weird edge cases when controlling APIs as opposed to hosts

    View Slide

  48. 48
    API Availability
    • es_instance_conn_validator doesn’t resolve until a connection can
    be made

    View Slide

  49. Some observations
    after a couple years…

    View Slide

  50. 50
    Results From the Field
    • One parent class makes
    creating more easy
    • Supported REST-based
    resources include:
    • indices
    • templates
    • pipelines
    • + more
    Extensibility
    • rspec + webmock for
    great testing
    • ES docs + specs first
    have made some
    implementations first try
    successes
    • Good mocks make some
    acceptance tests
    unnecessary (faster CI!)
    Reliability
    • Much easier to extend to
    new OS’s (i.e., Windows)
    • Greater control has made
    some tasks (like 3.x →
    4.x module update)
    smooth
    + more

    View Slide

  51. 51
    Questions?
    • github.com/elastic
    Thank You!

    View Slide