Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

F43919144cdcddd7ba50e46f71667d08?s=47 Tyler L
October 12, 2017

Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

F43919144cdcddd7ba50e46f71667d08?s=128

Tyler L

October 12, 2017
Tweet

Transcript

  1. Tyler Langlois, October 12th 2017 Software Engineer, Elastic @leothrix, github:

    tylerjl Custom Types and Providers: Modeling Modern REST Interfaces and Beyond
  2. 2 Obligatory “About Me” Slide • Been with company since

    2014 • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and Kibana) • Puppet-ing in one way or another over my whole professional career • Brought too many Elastic stickers that need to be given away (please partake) • Talk to me about Elasticsearch/Logstash/Kibana/Beats! Infrastructure/Operations/Software Engineer @ Elastic
  3. 3 Who is This Presentation For? Developers who work with

    Puppet modules Puppet users who want to dip into native type/provider development “What in the %@#$ is the Elasticsearch module doing” Operators who want to automate against APIs Hopefully empowers you to implement custom resources on your own
  4. 4 WHAT DOES THIS TALK’S TITLE EVEN MEAN?

  5. 5

  6. 6 Managing Resources with Raw APIs Example: CloudFormation • Pro:

    • Infrastructure resources are data • Extensible • Con: • Managing changes • Grokking huge chunks of JSON
  7. 7 Modeling Resources with Raw APIs Example: Terraform • Pro:

    • Readable • Manageable • Lifecycle + changes • Interoperability between other systems
  8. 8 Modeling Resources in Puppet A DSL to Model Disparate

    Resources A Graph to Manage Relationships A Concept of Changes to Manage Lifecycles ls, stat, chmod, chown sysv, systemd, upstart deb, rpm, pkg }
  9. 9 Modeling Resources in Puppet Abstraction is Powerful file {

    “/tmp/foo”: source => “puppet:///foo”, } -> package { “foo”: source => “/tmp/foo” } ~> service { “foo”: ensure => “running”, }
  10. 10 Modeling Resources in Puppet ? Elasticsearch Logstash Other REST

    APIs }
  11. 11 Modeling Resources in Puppet Extending the idea to APIs

    elasticsearch::template { “logstash”: content => { “template” => “*”, “settings” => { “number_of_replicas” => 0 } } } -> service { “es-app”: ensure => “running” }
  12. 12 Modeling Resources in Puppet • State Changes • Instead

    of comparing changes with GET responses and template files, compare during a no-op • A change in state can form dependencies and refresh events into other resources • Trickling changes up via reports lends better visibility Benefits
  13. 13 Modeling Resources in Puppet • State Changes • More

    fine-grained control • Most resources can be represented as Puppet hashes, so Hiera can be fully leveraged • Communicating via full Ruby HTTP libraries means CA files, auth, and more are easier to control • TESTS! Benefits
  14. 14 Modeling Resources in Puppet • State Changes • More

    fine-grained control • Some existing API-based resources: • Kubernetes module (swagger-generated) • Google Cloud • Following examples will be low-level (i.e. with just native Ruby HTTP libraries) • …hopefully, will help you write your own for $system
  15. 15 Let’s (briefly) talk about Puppet Types and Providers

  16. 16 Types, Providers, and their Resources Underlying Resource Puppet Provider

    Puppet Type • Has some way to change a property • Its state is introspectable and discoverable • Uniquely identified • How Ruby interacts with actual commands/system properties • Knows how to discover the properties of resources • Normalized provider API to Puppet DSL • Somewhat typed, catalog compilation • Abstraction over providers
  17. 17 Types, Providers, and their Resources: service • systemctl/service/rc commands

    • Startup visibility with enable/ chkconfig/etc. • Primarily shell-based for state • One provider for each init system • Ruby knows which shell commands to invoke to start, stop, enable, etc. • Unified API to start, enable, and restart a general service resource • Abstraction over provider- specific implementations • What we see in a manifest Underlying Resource Puppet Provider Puppet Type
  18. 18 Types, Providers, and their Resources: elasticsearch • REST API

    endpoints • Objects modeled in JSON • Individual endpoints via _template, _ingest, etc. • One provider base class, one provider per resource type • Using native Ruby HTTP APIs are high-level enough • Better alternative than `exec { “curl”:` • Resource properties expressed in Puppet DSL hashes • We don’t make API calls, we declare desired state Underlying Resource Puppet Provider Puppet Type
  19. 19 Then:

  20. 20 Now:

  21. Case Study: Elasticsearch Pipelines curl vs. Puppet

  22. 22 Ingest Pipelines

  23. 23 Ingest Pipelines

  24. 24 Ingest Pipelines • All pipelines are uniquely identified by

    a name (like defined or native types!) • Endpoints to manage pipelines: • GET to retrieve JSON object enumerating all pipelines • Note: can also retrieved based by name alone • PUT to create with JSON body • Note that we’re using unauthenticated APIs right now Key observations
  25. 25 Ingest Pipelines: Puppet Type

  26. 26 Ingest Pipelines: Puppet Type (Implementation)

  27. 27 Ingest Pipelines: Puppet Type (Implementation) …what the included abstraction

    does
  28. 28 Ingest Pipelines: Puppet Provider (Implementation)

  29. 29 Ingest Pipelines: Puppet Provider (details) …what the parent class

    does
  30. 30 Ingest Pipelines: Puppet Provider (details) …what the parent class

    does
  31. 31 Ingest Pipelines: Puppet Tests

  32. 32 Ingest Pipelines • That’s most of it! • Test-driven

    development + rspec makes it smooth • Bulk is abstracted; the beefy parts are in parent classes and reused by templates, indices, etc. • Native types and providers ≠ scary Summary
  33. 33 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch`

    Leveraging type-level tools HTTP API availability 1 2 3 4
  34. 34

  35. 35 An Example of Returning a Hash to Prefetch Automatically

    Gathering Resources uri = URI(“http://localhost:9200/_template”) http = Net::HTTP.new uri.host, uri.port req = Net::HTTP::Get.new uri.request_uri response = http.request req JSON.parse(response.body).map do |object_name, api_object| { :name => object_name, :ensure => :present, :content => api_object, :provider => name } end
  36. 36 Advantages • puppet resource functionality • Minimizes chatter with

    API endpoints • i.e., checking for existence versus properties, etc. • Call flush only when necessary • Additional API freebies (i.e., centralized access in flush(), etc.) Prefetching resources versus vanilla exists?
  37. 37

  38. 38 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch`

    Leveraging type-level tools HTTP API availability 1 2 3 4
  39. 39 Response Content vs. Request Content Usually never 1:1 mappings

    { "logstash": { "order": 0, "version": 60001, "index_patterns": [ "logstash-*" ], . . . elasticsearch::template { 'logstash': content => { 'template' => ‘logstash-*’, 'settings' => { . . . vs.
  40. 40 Types To the Rescue • A resource’s desired state

    is almost never the plain response for a query against the resource • Example: kubernetes Deployment versus the state of a Deployment • munge can help unify the resource versus JSON for comparability • insync? can be enhanced to understand which fields are being explicitly controlled by a user • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in {“another”: “field”} • Used pretty heavily in puppet-elasticsearch Managing response data
  41. 41 Example: Setting Default Fields Elasticsearch template # Set default

    values for templates munge do |value| { 'order' => 0, 'aliases' => {}, 'mappings' => {} }.merge(value) end
  42. 42 Example: Unifying Formatting Elasticsearch template # Normalize then compare

    the Puppet hash and json def insync?(is) Puppet_X::Elastic.deep_implode(is) == \ Puppet_X::Elastic.deep_implode(should) end { “foo”: { “bar”: “value” } } { “foo.bar”: “value” }
  43. 43 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch`

    Leveraging type-level tools HTTP API availability 1 2 3 4
  44. 44 HTTP In Providers

  45. 45 HTTP In Providers • Native HTTP libraries let us

    more easily control and pass: • TLS certificate authorities and verification booleans • HTTP basic auth credentials • Failure cases (timeouts, 4xx/5xx response codes, etc.) • In this case with Elasticsearch, error responses can return JSON messages for more helpful Puppet failures
  46. 46 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch`

    Leveraging type-level tools HTTP API availability 1 2 3 4
  47. 47 API Availability • What happens if: • An API-based

    REST resource requires an API to be up, not just a daemon? • A resource should block until one is available? • An unrelated resource needs that API as well? Weird edge cases when controlling APIs as opposed to hosts
  48. 48 API Availability • es_instance_conn_validator doesn’t resolve until a connection

    can be made
  49. Some observations after a couple years…

  50. 50 Results From the Field • One parent class makes

    creating more easy • Supported REST-based resources include: • indices • templates • pipelines • + more Extensibility • rspec + webmock for great testing • ES docs + specs first have made some implementations first try successes • Good mocks make some acceptance tests unnecessary (faster CI!) Reliability • Much easier to extend to new OS’s (i.e., Windows) • Greater control has made some tasks (like 3.x → 4.x module update) smooth + more
  51. 51 Questions? • github.com/elastic Thank You!