Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

Slide 1

Slide 1 text

Tyler Langlois, October 12th 2017 Software Engineer, Elastic @leothrix, github: tylerjl Custom Types and Providers: Modeling Modern REST Interfaces and Beyond

Slide 2

Slide 2 text

2 Obligatory “About Me” Slide • Been with company since 2014 • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and Kibana) • Puppet-ing in one way or another over my whole professional career • Brought too many Elastic stickers that need to be given away (please partake) • Talk to me about Elasticsearch/Logstash/Kibana/Beats! Infrastructure/Operations/Software Engineer @ Elastic

Slide 3

Slide 3 text

3 Who is This Presentation For? Developers who work with Puppet modules Puppet users who want to dip into native type/provider development “What in the %@#$ is the Elasticsearch module doing” Operators who want to automate against APIs Hopefully empowers you to implement custom resources on your own

Slide 4

Slide 4 text

4 WHAT DOES THIS TALK’S TITLE EVEN MEAN?

Slide 5

Slide 5 text

Slide 6

Slide 6 text

6 Managing Resources with Raw APIs Example: CloudFormation • Pro: • Infrastructure resources are data • Extensible • Con: • Managing changes • Grokking huge chunks of JSON

Slide 7

Slide 7 text

7 Modeling Resources with Raw APIs Example: Terraform • Pro: • Readable • Manageable • Lifecycle + changes • Interoperability between other systems

Slide 8

Slide 8 text

8 Modeling Resources in Puppet A DSL to Model Disparate Resources A Graph to Manage Relationships A Concept of Changes to Manage Lifecycles ls, stat, chmod, chown sysv, systemd, upstart deb, rpm, pkg }

Slide 9

Slide 9 text

9 Modeling Resources in Puppet Abstraction is Powerful file { “/tmp/foo”: source => “puppet:///foo”, } -> package { “foo”: source => “/tmp/foo” } ~> service { “foo”: ensure => “running”, }

Slide 10

Slide 10 text

10 Modeling Resources in Puppet ? Elasticsearch Logstash Other REST APIs }

Slide 11

Slide 11 text

11 Modeling Resources in Puppet Extending the idea to APIs elasticsearch::template { “logstash”: content => { “template” => “*”, “settings” => { “number_of_replicas” => 0 } } } -> service { “es-app”: ensure => “running” }

Slide 12

Slide 12 text

12 Modeling Resources in Puppet • State Changes • Instead of comparing changes with GET responses and template files, compare during a no-op • A change in state can form dependencies and refresh events into other resources • Trickling changes up via reports lends better visibility Benefits

Slide 13

Slide 13 text

13 Modeling Resources in Puppet • State Changes • More fine-grained control • Most resources can be represented as Puppet hashes, so Hiera can be fully leveraged • Communicating via full Ruby HTTP libraries means CA files, auth, and more are easier to control • TESTS! Benefits

Slide 14

Slide 14 text

14 Modeling Resources in Puppet • State Changes • More fine-grained control • Some existing API-based resources: • Kubernetes module (swagger-generated) • Google Cloud • Following examples will be low-level (i.e. with just native Ruby HTTP libraries) • …hopefully, will help you write your own for $system

Slide 15

Slide 15 text

15 Let’s (briefly) talk about Puppet Types and Providers

Slide 16

Slide 16 text

16 Types, Providers, and their Resources Underlying Resource Puppet Provider Puppet Type • Has some way to change a property • Its state is introspectable and discoverable • Uniquely identified • How Ruby interacts with actual commands/system properties • Knows how to discover the properties of resources • Normalized provider API to Puppet DSL • Somewhat typed, catalog compilation • Abstraction over providers

Slide 17

Slide 17 text

17 Types, Providers, and their Resources: service • systemctl/service/rc commands • Startup visibility with enable/ chkconfig/etc. • Primarily shell-based for state • One provider for each init system • Ruby knows which shell commands to invoke to start, stop, enable, etc. • Unified API to start, enable, and restart a general service resource • Abstraction over provider- specific implementations • What we see in a manifest Underlying Resource Puppet Provider Puppet Type

Slide 18

Slide 18 text

18 Types, Providers, and their Resources: elasticsearch • REST API endpoints • Objects modeled in JSON • Individual endpoints via _template, _ingest, etc. • One provider base class, one provider per resource type • Using native Ruby HTTP APIs are high-level enough • Better alternative than `exec { “curl”:` • Resource properties expressed in Puppet DSL hashes • We don’t make API calls, we declare desired state Underlying Resource Puppet Provider Puppet Type

Slide 19

Slide 19 text

19 Then:

Slide 20

Slide 20 text

20 Now:

Slide 21

Slide 21 text

Case Study: Elasticsearch Pipelines curl vs. Puppet

Slide 22

Slide 22 text

22 Ingest Pipelines

Slide 23

Slide 23 text

23 Ingest Pipelines

Slide 24

Slide 24 text

24 Ingest Pipelines • All pipelines are uniquely identified by a name (like defined or native types!) • Endpoints to manage pipelines: • GET to retrieve JSON object enumerating all pipelines • Note: can also retrieved based by name alone • PUT to create with JSON body • Note that we’re using unauthenticated APIs right now Key observations

Slide 25

Slide 25 text

25 Ingest Pipelines: Puppet Type

Slide 26

Slide 26 text

26 Ingest Pipelines: Puppet Type (Implementation)

Slide 27

Slide 27 text

27 Ingest Pipelines: Puppet Type (Implementation) …what the included abstraction does

Slide 28

Slide 28 text

28 Ingest Pipelines: Puppet Provider (Implementation)

Slide 29

Slide 29 text

29 Ingest Pipelines: Puppet Provider (details) …what the parent class does

Slide 30

Slide 30 text

30 Ingest Pipelines: Puppet Provider (details) …what the parent class does

Slide 31

Slide 31 text

31 Ingest Pipelines: Puppet Tests

Slide 32

Slide 32 text

32 Ingest Pipelines • That’s most of it! • Test-driven development + rspec makes it smooth • Bulk is abstracted; the beefy parts are in parent classes and reused by templates, indices, etc. • Native types and providers ≠ scary Summary

Slide 33

Slide 33 text

33 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4

Slide 34

Slide 34 text

Slide 35

Slide 35 text

35 An Example of Returning a Hash to Prefetch Automatically Gathering Resources uri = URI(“http://localhost:9200/_template”) http = Net::HTTP.new uri.host, uri.port req = Net::HTTP::Get.new uri.request_uri response = http.request req JSON.parse(response.body).map do |object_name, api_object| { :name => object_name, :ensure => :present, :content => api_object, :provider => name } end

Slide 36

Slide 36 text

36 Advantages • puppet resource functionality • Minimizes chatter with API endpoints • i.e., checking for existence versus properties, etc. • Call flush only when necessary • Additional API freebies (i.e., centralized access in flush(), etc.) Prefetching resources versus vanilla exists?

Slide 37

Slide 37 text

Slide 38

Slide 38 text

38 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4

Slide 39

Slide 39 text

39 Response Content vs. Request Content Usually never 1:1 mappings { "logstash": { "order": 0, "version": 60001, "index_patterns": [ "logstash-*" ], . . . elasticsearch::template { 'logstash': content => { 'template' => ‘logstash-*’, 'settings' => { . . . vs.

Slide 40

Slide 40 text

40 Types To the Rescue • A resource’s desired state is almost never the plain response for a query against the resource • Example: kubernetes Deployment versus the state of a Deployment • munge can help unify the resource versus JSON for comparability • insync? can be enhanced to understand which fields are being explicitly controlled by a user • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in {“another”: “field”} • Used pretty heavily in puppet-elasticsearch Managing response data

Slide 41

Slide 41 text

41 Example: Setting Default Fields Elasticsearch template # Set default values for templates munge do |value| { 'order' => 0, 'aliases' => {}, 'mappings' => {} }.merge(value) end

Slide 42

Slide 42 text

42 Example: Unifying Formatting Elasticsearch template # Normalize then compare the Puppet hash and json def insync?(is) Puppet_X::Elastic.deep_implode(is) == \ Puppet_X::Elastic.deep_implode(should) end { “foo”: { “bar”: “value” } } { “foo.bar”: “value” }

Slide 43

Slide 43 text

43 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4

Slide 44

Slide 44 text

44 HTTP In Providers

Slide 45

Slide 45 text

45 HTTP In Providers • Native HTTP libraries let us more easily control and pass: • TLS certificate authorities and verification booleans • HTTP basic auth credentials • Failure cases (timeouts, 4xx/5xx response codes, etc.) • In this case with Elasticsearch, error responses can return JSON messages for more helpful Puppet failures

Slide 46

Slide 46 text

46 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4

Slide 47

Slide 47 text

47 API Availability • What happens if: • An API-based REST resource requires an API to be up, not just a daemon? • A resource should block until one is available? • An unrelated resource needs that API as well? Weird edge cases when controlling APIs as opposed to hosts

Slide 48

Slide 48 text

48 API Availability • es_instance_conn_validator doesn’t resolve until a connection can be made

Slide 49

Slide 49 text

Some observations after a couple years…

Slide 50

Slide 50 text

50 Results From the Field • One parent class makes creating more easy • Supported REST-based resources include: • indices • templates • pipelines • + more Extensibility • rspec + webmock for great testing • ES docs + specs first have made some implementations first try successes • Good mocks make some acceptance tests unnecessary (faster CI!) Reliability • Much easier to extend to new OS’s (i.e., Windows) • Greater control has made some tasks (like 3.x → 4.x module update) smooth + more

Slide 51

Slide 51 text

51 Questions? • github.com/elastic Thank You!