Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

1. Tyler Langlois, October 12th 2017 Software Engineer, Elastic @leothrix, github: tylerjl Custom Types and Providers: Modeling Modern REST Interfaces and Beyond
2. 2 Obligatory “About Me” Slide • Been with company since 2014 • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and Kibana) • Puppet-ing in one way or another over my whole professional career • Brought too many Elastic stickers that need to be given away (please partake) • Talk to me about Elasticsearch/Logstash/Kibana/Beats! Infrastructure/Operations/Software Engineer @ Elastic
3. 3 Who is This Presentation For? Developers who work with Puppet modules Puppet users who want to dip into native type/provider development “What in the %@#$ is the Elasticsearch module doing” Operators who want to automate against APIs Hopefully empowers you to implement custom resources on your own
4. 4 WHAT DOES THIS TALK’S TITLE EVEN MEAN?
6. 6 Managing Resources with Raw APIs Example: CloudFormation • Pro: • Infrastructure resources are data • Extensible • Con: • Managing changes • Grokking huge chunks of JSON
7. 7 Modeling Resources with Raw APIs Example: Terraform • Pro: • Readable • Manageable • Lifecycle + changes • Interoperability between other systems
8. 8 Modeling Resources in Puppet A DSL to Model Disparate Resources A Graph to Manage Relationships A Concept of Changes to Manage Lifecycles ls, stat, chmod, chown sysv, systemd, upstart deb, rpm, pkg }
9. 9 Modeling Resources in Puppet Abstraction is Powerful file { “/tmp/foo”: source => “puppet:///foo”, } -> package { “foo”: source => “/tmp/foo” } ~> service { “foo”: ensure => “running”, }
10. 10 Modeling Resources in Puppet ? Elasticsearch Logstash Other REST APIs }
11. 11 Modeling Resources in Puppet Extending the idea to APIs elasticsearch::template { “logstash”: content => { “template” => “*”, “settings” => { “number_of_replicas” => 0 } } } -> service { “es-app”: ensure => “running” }
12. 12 Modeling Resources in Puppet • State Changes • Instead of comparing changes with GET responses and template files, compare during a no-op • A change in state can form dependencies and refresh events into other resources • Trickling changes up via reports lends better visibility Benefits
13. 13 Modeling Resources in Puppet • State Changes • More fine-grained control • Most resources can be represented as Puppet hashes, so Hiera can be fully leveraged • Communicating via full Ruby HTTP libraries means CA files, auth, and more are easier to control • TESTS! Benefits
14. 14 Modeling Resources in Puppet • State Changes • More fine-grained control • Some existing API-based resources: • Kubernetes module (swagger-generated) • Google Cloud • Following examples will be low-level (i.e. with just native Ruby HTTP libraries) • …hopefully, will help you write your own for $system
15. 15 Let’s (briefly) talk about Puppet Types and Providers
16. 16 Types, Providers, and their Resources Underlying Resource Puppet Provider Puppet Type • Has some way to change a property • Its state is introspectable and discoverable • Uniquely identified • How Ruby interacts with actual commands/system properties • Knows how to discover the properties of resources • Normalized provider API to Puppet DSL • Somewhat typed, catalog compilation • Abstraction over providers
17. 17 Types, Providers, and their Resources: service • systemctl/service/rc commands • Startup visibility with enable/ chkconfig/etc. • Primarily shell-based for state • One provider for each init system • Ruby knows which shell commands to invoke to start, stop, enable, etc. • Unified API to start, enable, and restart a general service resource • Abstraction over provider- specific implementations • What we see in a manifest Underlying Resource Puppet Provider Puppet Type
18. 18 Types, Providers, and their Resources: elasticsearch • REST API endpoints • Objects modeled in JSON • Individual endpoints via _template, _ingest, etc. • One provider base class, one provider per resource type • Using native Ruby HTTP APIs are high-level enough • Better alternative than `exec { “curl”:` • Resource properties expressed in Puppet DSL hashes • We don’t make API calls, we declare desired state Underlying Resource Puppet Provider Puppet Type
19. 19 Then:
20. 20 Now:
21. Case Study: Elasticsearch Pipelines curl vs. Puppet
22. 22 Ingest Pipelines
23. 23 Ingest Pipelines
24. 24 Ingest Pipelines • All pipelines are uniquely identified by a name (like defined or native types!) • Endpoints to manage pipelines: • GET to retrieve JSON object enumerating all pipelines • Note: can also retrieved based by name alone • PUT to create with JSON body • Note that we’re using unauthenticated APIs right now Key observations
25. 25 Ingest Pipelines: Puppet Type
26. 26 Ingest Pipelines: Puppet Type (Implementation)
27. 27 Ingest Pipelines: Puppet Type (Implementation) …what the included abstraction does
28. 28 Ingest Pipelines: Puppet Provider (Implementation)
29. 29 Ingest Pipelines: Puppet Provider (details) …what the parent class does
30. 30 Ingest Pipelines: Puppet Provider (details) …what the parent class does
31. 31 Ingest Pipelines: Puppet Tests
32. 32 Ingest Pipelines • That’s most of it! • Test-driven development + rspec makes it smooth • Bulk is abstracted; the beefy parts are in parent classes and reused by templates, indices, etc. • Native types and providers ≠ scary Summary
33. 33 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
35. 35 An Example of Returning a Hash to Prefetch Automatically Gathering Resources uri = URI(“http://localhost:9200/_template”) http = Net::HTTP.new uri.host, uri.port req = Net::HTTP::Get.new uri.request_uri response = http.request req JSON.parse(response.body).map do |object_name, api_object| { :name => object_name, :ensure => :present, :content => api_object, :provider => name } end
36. 36 Advantages • puppet resource functionality • Minimizes chatter with API endpoints • i.e., checking for existence versus properties, etc. • Call flush only when necessary • Additional API freebies (i.e., centralized access in flush(), etc.) Prefetching resources versus vanilla exists?
38. 38 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
39. 39 Response Content vs. Request Content Usually never 1:1 mappings { "logstash": { "order": 0, "version": 60001, "index_patterns": [ "logstash-*" ], . . . elasticsearch::template { 'logstash': content => { 'template' => ‘logstash-*’, 'settings' => { . . . vs.
40. 40 Types To the Rescue • A resource’s desired state is almost never the plain response for a query against the resource • Example: kubernetes Deployment versus the state of a Deployment • munge can help unify the resource versus JSON for comparability • insync? can be enhanced to understand which fields are being explicitly controlled by a user • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in {“another”: “field”} • Used pretty heavily in puppet-elasticsearch Managing response data
41. 41 Example: Setting Default Fields Elasticsearch template # Set default values for templates munge do |value| { 'order' => 0, 'aliases' => {}, 'mappings' => {} }.merge(value) end
42. 42 Example: Unifying Formatting Elasticsearch template # Normalize then compare the Puppet hash and json def insync?(is) Puppet_X::Elastic.deep_implode(is) == \ Puppet_X::Elastic.deep_implode(should) end { “foo”: { “bar”: “value” } } { “foo.bar”: “value” }
43. 43 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
44. 44 HTTP In Providers
45. 45 HTTP In Providers • Native HTTP libraries let us more easily control and pass: • TLS certificate authorities and verification booleans • HTTP basic auth credentials • Failure cases (timeouts, 4xx/5xx response codes, etc.) • In this case with Elasticsearch, error responses can return JSON messages for more helpful Puppet failures
46. 46 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
47. 47 API Availability • What happens if: • An API-based REST resource requires an API to be up, not just a daemon? • A resource should block until one is available? • An unrelated resource needs that API as well? Weird edge cases when controlling APIs as opposed to hosts
48. 48 API Availability • es_instance_conn_validator doesn’t resolve until a connection can be made
49. Some observations after a couple years…
50. 50 Results From the Field • One parent class makes creating more easy • Supported REST-based resources include: • indices • templates • pipelines • + more Extensibility • rspec + webmock for great testing • ES docs + specs first have made some implementations first try successes • Good mocks make some acceptance tests unnecessary (faster CI!) Reliability • Much easier to extend to new OS’s (i.e., Windows) • Greater control has made some tasks (like 3.x → 4.x module update) smooth + more
51. 51 Questions? • github.com/elastic Thank You!