Creating a Terraform Provider for Just About Anything

Terrafom is an amazing tool that lets you define your infrastructure as code. Under the hood it's an incredibly powerful state machine that makes API requests and marshals resources. In this talk we'll dive into the inner workings of Terraform and examine all the elements of a provider — from the documentation to the test suite. You'll walk away with the knowledge of how to contribute to an existing provider or create your own from scratch. We'll also take a look at some of the things we encountered while working on the DigitalOcean provider and the lessons learned from the community.

https://youtu.be/noxwUVet5RE

Transcript

1. Creating a Terraform Provider for Just about Anything Eddie Zaneski @eddiezane @DigitalOcean
2. Terraform is?
3. Getty Images/Shutterstock/NASA; illustration by Dave Mosher/Business Insider https://www.businessinsider.com/elon-musk-mars-colony-details-new-space-study-2017-6
4. Terraform is?
5. Terraform is a state machine
6. Terraform Provider?
8. provider "digitalocean" { token = "DIGITALOCEAN_API_TOKEN" } resource "digitalocean_droplet" "web" { name = "web" region = "sfo1" size = "s-1vcpu-1gb" image = "ubuntu-18-04-x64" }
9. provider "kubernetes" { config_context_auth_info = "ops" config_context_cluster = "mycluster" } resource "kubernetes_service" "dope_web_server" { metadata { name = "dope_web_server" labels = { app = "web" } } spec { type = "LoadBalancer" selector { app = "${kubernetes_pod.web.metadata.0.labels.app}" } port { port = 8080 target_port = 80 } } }
10. provider "github" { token = "${var.github_token}" organization = "${var.github_organization}" } resource "github_repository" "codez" { name = "uber-for-tacos" description = "My new startup" private = true }
11. provider "googlecalendar" {} resource "googlecalendar_event" "talk" { summary = "This Terraform talk" location = "Gold Room" start = "2018-10-23T14:35:00-08:00" end = "2018-10-23T15:10:00-08:00" attendee { email = "eddiezane@digitalocean.com" } } @sethvargo https://www.hashicorp.com/blog/managing-google-calendar-with-terraform
12. provider "hue" { token = "YOUR_HUE_TOKEN" bridge_ip = "192.168.1.137" } data "hue_light" "kitchen" { light_id = "1" } resource "hue_light_state" "kitchen" { light_id = "${data.hue_light.kitchen.light_id}" brightness = 42 red = 246 blue = 74 green = 138 }
13. provider "todoist" { api_key = "TODOIST_API_KEY" } resource "todoist_task" "pack" { content = "Pack to move to Denver" due_string = "Friday" responsible_uid = "my_wife" completed = false } https://github.com/eddiezane/terraform-provider-todoist https://github.com/eddiezane/todoist-rest-go
15. Build a Provider for?
16. Your Company ◦ You're a cloud provider ◦ Your customers ask for one ◦ You have a resource in an API that makes sense to manage with Terraform ◦ Skip a front end ◦ Don't need to write duct tape apps ◦ Internal
17. terraform-provider-dointernal ◦ Very shallow shim/fork ◦ Additional features/fields ◦ Hypervisor placement ◦ Register into Chef Server ◦ Expose beta features
18. Contribute to a Provider
19. Contribute ◦ Fix a bug ◦ Add a feature ◦ Hacktoberfest!
20. Maintain a Fork ◦ Org specific defaults or needs ◦ Missing features ◦ Unresponsive maintainer
21. Convention
22. Where to Start?
23. Go API Client ◦ Separate your API logic from your provider logic ◦ Error handling ◦ Logging
24. Skeleton/Guide package todoist import ( todoistRest "github.com/eddiezane/todoist-rest-go" "github.com/hashicorp/terraform/helper/schema" "github.com/hashicorp/terraform/terraform" ) func Provider() terraform.ResourceProvider { https://www.terraform.io/docs/extend/writing-custom-providers.html
25. Read other Providers
26. return &schema.Provider{ Schema: map[string]*schema.Schema{ "api_key": &schema.Schema{ Type: schema.TypeString, Description: "Your Todoist API key", Required: true, DefaultFunc: schema.EnvDefaultFunc("TODOIST_API_KEY", nil), }, }, ConfigureFunc: configureFunc(), ... } Schema o Used in all the things o Type enforcement
27. Types ◦ TypeBool ◦ TypeInt ◦ TypeFloat ◦ TypeString ◦ TypeList ◦ TypeMap ◦ TypeSet
28. "Terraform stdlib" ◦ https://github.com/hashicorp/terraform/tree/master/helper ◦ Read it! ◦ schema ◦ Provider ◦ Resource ◦ ResourceData ◦ Validators ◦ IntBetween ◦ Testing ◦ RandString(strlen int)
29. func configureFunc() func(*schema.ResourceData) (interface{}, error) { return func(d *schema.ResourceData) (interface{}, error) { client := todoistRest.NewClient(d.Get("api_key").(string)) return client, nil } } ConfigureFunc (meta) o Return the meta
30. ResourcesMap: map[string]*schema.Resource{ "todoist_task": resourceTodoistTask(), "todoist_project": resourceTodoistProject(), }, Resources o One file per resource o resource_todoist_task.go
31. func resourceTodoistTask() *schema.Resource { return &schema.Resource{ Schema: map[string]*schema.Schema{ "content": &schema.Schema{ Type: schema.TypeString, Required: true, }, "completed": &schema.Schema{ Type: schema.TypeBool, Optional: true, Default: false, }, }, Create: resourceTodoistTaskCreate, Read: resourceTodoistTaskRead, Update: resourceTodoistTaskUpdate, Delete: resourceTodoistTaskDelete, } }
32. func resourceTodoistTaskCreate(d *schema.ResourceData, meta interface{}) error { client := meta.(*todoistRest.Client) content := d.Get("content").(string) newTask := &todoistRest.NewTask{ Content: content, } task, err := client.CreateTask(newTask) if err != nil { return err } d.SetId(task.Id) return resourceTodoistTaskRead(d, meta) }
33. func resourceTodoistTaskRead(d *schema.ResourceData, meta interface{}) error { client := meta.(*todoistRest.Client) id := d.Id() task, err := client.GetTask(id) if err != nil { return err } ... d.Set("content", t.Content) d.Set("completed", t.Completed) return nil }
34. func resourceTodoistTaskUpdate(d *schema.ResourceData, meta interface{}) error { client := meta.(*todoistRest.Client) id := d.Id() if d.HasChange("content") { content := d.Get("content").(string) err := client.UpdateTask(... } return nil }
35. func resourceTodoistTaskDelete(d *schema.ResourceData, meta interface{}) error { client := meta.(*todoistRest.Client) id := d.Id() err := client.DeleteTask(id) if err != nil { return err } return nil }
36. ◦ Extremely robust to failures and fault tolerant ◦ Log all the things! ◦ TF_LOG=INFO terraform plan ◦ Quickly identify what's your API's fault vs Terraform vs random ◦ Partial state If the Create callback returns with or without an error and an ID has been set, the resource is assumed created and all state is saved with it. Error Handling https://www.terraform.io/docs/extend/writing-custom-providers.html#error-handling-amp-partial-state
37. Testing
38. func TestAccTodoistTask_basic(t *testing.T) { var task todoistRest.Task content := "test task content" resource.Test(t, resource.TestCase{ PreCheck: func() { testAccPreCheck(t) }, Providers: testAccProviders, CheckDestroy: testAccCheckTodoistTaskDestroy, Steps: []resource.TestStep{ { Config: testAccCheckTodoistTaskConfig_basic(content), Check: resource.ComposeTestCheckFunc( testAccCheckTodoistTaskExists("todoist_task.test", &task), testAccCheckTodoistTaskAttributes(&task, content), resource.TestCheckResourceAttr("todoist_task.test", "content", content), ), }, }, }) }
39. func testAccCheckTodoistTaskConfig_basic(content string) string { return fmt.Sprintf(` resource "todoist_task" "test" { content = "%s" } `, content) }
40. ◦ Precheck > Test Steps > Destroy > CheckDestroy ◦ One config per step ◦ Use steps to simulate updates and deletes ◦ Reuse as much as possible—especially early on ◦ Makefile ◦ make testacc TESTARGS='-run=TestAccDigitalOceanDomain_Basic' Testing
41. Docs ◦ Great place to get started ◦ Magic `website` folder
42. Process Notes ◦ Engage HashiCorp early ◦ Travis CI for CI/CD ◦ Releases via Slack ◦ TeamCity for releasing https://www.terraform.io/guides/terraform-provider-development-program.html
43. Tips ◦ Learn a bit of Go first ◦ Get familiar with `strconv` ◦ Custom `UnmarshalJSON` functions ◦ Have a solid Go API client library ◦ Fix design problems with your API if possible ◦ Understand Terraform Modules ◦ Don't be afraid to copy/pasta ◦ Focus on composability and reusable functions ◦ Sweepers https://www.terraform.io/guides/terraform-provider-development-program.html
44. Resources ◦ Other providers ◦ Terraform docs, guides, and source code ◦ Videos ◦ How to Extend the Terraform Provider List - Paul Stack ▪ https://youtu.be/2BvpqmFpchI ◦ Terraform and The Extensible Provider Architecture - Clint Shryock ▪ https://youtu.be/TMmovxyo5sY ◦ OpenCredo
45. Thanks! @eddiezane @DigitalOcean https://do.co/hashiconf-2018-terraform