Tricks & Treats with HashiCorps OSS

Transcript

1. Tricks & Treats with HashiCorp’s OSS Allee Clark Site Reliability Engineer @ Oscar Health
2. Open Source Software (OSS) HashiCorp writes a lot of software for the community What treats for today Go-Getter and Consul template
3. Overview of today 1. Go-Getter 2. Consul-template 3. Other Open source projects
4. Go-Getter github.com/hashicorp/go-getter Copyright © 2020 HashiCorp
5. TERMINAL > go-getter github.com/foo/bar ./foo
6. Protocols and Detectors ▪ Local files ▪ Git ▪ Mercurial ▪ HTTP ▪ Amazon S3 ▪ Google GCP
7. Forced protocols "http://github.com/mitchellh/vagrant.git" or git::http://github.com/mitchellh/vagrant.git
8. How Git works Clones It clones the repository in memory and discard everything that isn’t needed.
9. How Cloud Storage works Leverages 3rd Party Uses Amazon’s or Google’s storage API to optimize operations.
10. How would I use go-getter?
11. Detector interface CODE EDITOR // Detector defines the interface that an invalid URL or a URL with a blank // scheme is passed through in order to determine if its shorthand for // something else well-known. type Detector interface { // Detect will detect whether the string matches a known pattern to // turn it into a proper URL. Detect(string, string) (string, bool, error) }
12. Detector CODE EDITOR //define the type of detectors to use -- in this case only github is needed detector := []getter.Detector{new(getter.GitHubDetector)}
13. Getter interface CODE EDITOR // Getter defines the interface that schemes must implement to download type Getter interface { // Get downloads the given URL into the given directory. This always // assumes that we're updating and gets the latest version that it can. // // The directory may already exist (if we're updating). If it is in a // format that isn't understood, an error should be returned. Get shouldn't // simply nuke the directory. Get(string, *url.URL) error // GetFile downloads the give URL into the given path. The URL must // reference a single file. If possible, the Getter should check if // the remote end contains the same file and no-op this operation. GetFile(string, *url.URL) error
14. Getter interface CODE EDITOR // ClientMode returns the mode based on the given URL. This is used to // allow clients to let the getters decide which mode to use. ClientMode(*url.URL) (ClientMode, error) // SetClient allows a getter to know it's client // in order to access client's Get functions or // progress tracking. SetClient(*Client) }
15. Getter use case CODE EDITOR //provide the getter needed to download the files getters := map[string]getter.Getter{ "git": &getter.GitGetter{}}
16. Create a new client CODE EDITOR client := &getter.Client{ Ctx: context.Background(), //define the destination to where the directory will be stored. This will create the directory if it doesnt exist Dst: "/tmp/gogetter", Dir: true, //the repository with a subdirectory I would like to clone only Src: "github.com/hashicorp/terraform/examples/cross-provider", Mode: getter.ClientModeDir, }
17. Download the files CODE EDITOR if err := client.Get(); err != nil { fmt.Fprintf(os.Stderr, "Error getting path %s: %v", client.Src, err) os.Exit(1) }
19. 1000 foot view
21. Service Registration
22. Check system
23. Key/Value System
24. consul-template github.com/hashicorp/consul-template Copyright © 2019 HashiCorp
25. Where you can get your values from Vault Read secret values, static or dynamic from Vault paths and export them into your configuration. Consul Allows you to read values from Consul key and services into a file
26. TERMINAL > consul-template -template "in.tpl:out.txt" -once Design to be a daemon
27. CODE EDITOR {{ file "/usr/local/bin/foo" }} //renders bar
28. TERMINAL [root@e63609e6e248 tmpl]# ls bar.tpl foo [root@e63609e6e248 tmpl]# consul-template -template "bar.tpl:bar.txt" -once [root@e63609e6e248 tmpl]# cat bar.txt testdata [root@e63609e6e248 tmpl]# cat foo testdata
29. CODE EDITOR {{ key "service/redis/maxconns" }} //renders 15
30. Template Let’s run a simple example of consul template through code. CODE EDITOR consulTemplateInput := &template.NewTemplateInput{Source: “/foo”} templ, err := template.NewTemplate(consulTemplateInput) if err != nil { return err } response, err := templ.Execute(&template.ExecuteInput{}) if err != nil { return err } //do something with output fmt.Fprintln(os.Stdout, response.Out)
31. NewTemplateInput ▪ Source or Contents ▪ ErrMissingKey (bool) ▪ LeftDelim or RightDelim (string) ▪ FunctionBlacklist (array of strings) ▪ SandboxPath (array of strings)
32. ExecuteInput ▪ Brain (object of type Brain) ▪ Env (array of strings)
33. Advanced Usage
34. Configuration ▪ Consul (object of type ConsulConfig) ▪ Exec (map) ▪ KillSignal (type os.Signal) ▪ PidFile (string) ▪ ReloadSignal (type os.Signal) ▪ Syslog (type SyslogConfig) ▪ Vault (object of type VaultConfig)
35. Template multiple values TERMINAL > consul-template \ -template "/tmp/nginx.ctmpl:/var/nginx/nginx.conf:nginx -s reload" \ -template "/tmp/redis.ctmpl:/var/redis/redis.conf:service redis restart" \ -template "/tmp/haproxy.ctmpl:/var/haproxy/haproxy.conf"
36. Template Configuration ▪ Backup (bool) ▪ Contents (string) ▪ ErrMissingKey (bool) ▪ FunctionBlacklist (array of stings) ▪ Sandbox (array of strings)
37. Initialize manager CODE EDITOR // NewRunner accepts a slice of TemplateConfigs and returns a pointer to the new // Runner and any error that occurred during creation. runner, err := manger.NewRunner(defaultConfig, false) if err != nil { return err } var out bytes.Buffer go runner.Start() defer runner.Stop() if err = runner.Run(); err != nil { return err } runner.SetOutStream(&out) runner.SetErrStream(&out)
38. Wait for the runner to done or error Each as its own type. We want to save the result of the error so we can return that back to the client. The final case is to keep rendering the template CODE EDITOR for { select { case err := <-runner.ErrCh: return err case <-runner.DoneCh: return nil case <-runner.TemplateRenderedCh(): } }
39. Plugins
40. Authoring Plugins “For some use cases, it may be necessary to write a plugin that offloads work to another system. This is especially useful for things that may not fit in the "standard library" of Consul Template, but still need to be shared across multiple instances.”
41. NAME NAME - the name of the plugin - this is also the name of the binary, either a full path or just the program name. It will be executed in a shell with the inherited PATH so e.g. the plugin cat will run the first executable cat that is found on the PATH.
42. INPUT INPUT - input from the template. There will be one INPUT for every argument passed to the plugin function. If the arguments
43. CODE EDITOR ec2-consul-template-plugin.py metadata = boto.utils.get_instance_metadata() metadata['user_data'] = boto.utils.get_instance_userdata() metadata['identity'] = boto.utils.get_instance_identity() metadata['placement']['region'] = metadata['identity']['document']['region'] metadata = mangle_keys(metadata) if(len(sys.argv) > 1): if(len(sys.argv) > 2): output = {} for arg in sys.argv[1:]: output[arg] = xpath_get(metadata, arg) print json.dumps(output) else: if isinstance(xpath_get(metadata, sys.argv[1]), basestring): print xpath_get(metadata, sys.argv[1]) else: print json.dumps(xpath_get(metadata, sys.argv[1])) else: print json.dumps(metadata)
44. CODE EDITOR {{ plugin "ec2-consul-template-plugin.py" "placement/region" }}
45. Final thoughts
46. Other 7 Open source projects 1. serf 2. consul-replicate 3. consul-esm 4. envconsul 5. go-discover 6. go-memdb 7. memberlist
47. Thank You! alleeclark@gmail.com @alleeclark allee.xyz 47