How to build SDKs in Go

Lessons from building the Go SDK for Dropbox.
Slides from Golang UK 2017

Transcript

1. How to build SDKs in Go Lessons from the Go SDK for Dropbox
2. Diwaker Gupta • Infrastructure @ Dropbox • Metadata Services, Multihoming, Search, Messaging, Email, Notifications, Machine Learning etc • Authored Dropbox Go SDK and dbxcli • Twitter/Github: @diwakergupta
3. users businesses
4. • Gophercon ‘17 Keynote by @tammybutow: https://bit.ly/gophercon17dbx • Dozens of Go services, many with millions of RPS! • We open source at: https://github.com/dropbox • We are hiring! dropbox.com/jobs or email me (diwaker@)
5. Lessons from the Dropbox Go SDK
6. #1: Use a Code Generator
7. #1: Use a code generator • Language-agnostic API definition (DSL, JSON, YAML, etc), Language-specific generators • Dropbox Stone (spec + generators) ◦ https://github.com/dropbox/stone • Google ◦ JSON Spec https://developers.google.com/discovery/ ◦ Codegen https://github.com/google/google-api-go-client/ • AWS Go SDK has a very similar setup • OpenAPI: https://github.com/OAI/OpenAPI-Specification
8. #2: Avoid Surprises
9. #2: Avoid Surprises • Principle of Least Surprise • No external dependencies ◦ Only standard library imports • No vendored dependencies • What you need is what you get ◦ Scoped sub-packages for larger APIs (e.g. AWS) ◦ Make it `go get` friendly
10. #3: Make Configuration Simple
11. #3: Make Configuration Simple • Don’t use cmdline flags ◦ Stdlib limitations • Environment variables ◦ OK, but beware pitfalls • Use a config struct ◦ Same pattern in AWS SDK • Persistence outside SDK ◦ Let applications choose ◦ dbxcli load/stores json configs // Config contains parameters for configuring the SDK. type Config struct { // OAuth2 access token Token string // Enable verbose logging in SDK Verbose bool ... // For testing only Client *http.Client }
12. #4: Provide Visibility
13. #4: Provide Visibility • Allow verbose logging • Allow configurable logging targets • Limited by stdlib :( type Config struct { // Enable verbose logging Verbose bool // Optional logging target Logger *log.Logger } // TryLog will, if Verbose is set, log to // the config's logger or the default logger // if Config.Logger is nil. func (c *Config) TryLog(format string, v ...interface{}) { if !c.Verbose { return } if c.Logger != nil { c.Logger.Printf(format, v...) } else { log.Printf(format, v...) } }
14. #5: Use Go idioms for unsupported types
15. #5: Unions Consider (in Stone): union DeleteError path_lookup LookupError path_write WriteError In JSON (note LookupError): { ".tag": "path_lookup", "path_lookup": { ".tag": "malformed_path", "malformed_path": "/some/path" } } Equivalent struct in Go type DeleteError struct { dropbox.Tagged PathLookup *LookupError `json:"path_lookup"` PathWrite *WriteError `json:"path_write"` } Problem: (de)serialization
16. func (u *DeleteError) UnmarshalJSON(body []byte) error { type wrap struct { dropbox.Tagged PathLookup json.RawMessage PathWrite json.RawMessage } var w wrap switch w.Tag { case "path_lookup": err = json.Unmarshal(w.PathLookup, &u.PathLookup) ... case "path_write": err = json.Unmarshal(w.PathWrite, &u.PathWrite) ... } #5: Unions type DeleteError struct { dropbox.Tagged PathLookup *LookupError `json:"path_lookup,omitempty"` PathWrite *WriteError `json:"path_write,omitempty"` } { ".tag": "path_lookup", "path_lookup": { ".tag": "malformed_path", "malformed_path": "/some/path" } }
17. #5: Inherited Types In Stone struct Metadata // Common fields: Name, Path etc union file FileMetadata folder FolderMetadata Idiomatic Go: Embedding type Metadata struct { // Common fields } type FileMetadata struct { Metadata // FileMetadata specific fields } type FolderMetadata struct { Metadata // FolderMetadata specific fields }
18. Solution: Use a dummy interface type IsMetadata interface { IsMetadata() } // Subtypes get this via embedding func (u *Metadata) IsMetadata() {} // Use IsMetadata where you need // the union type #5: Inherited Types Problem: Polymorphism func List(...) []Metadata Can return FileMetadata or FolderMetadata
19. Similar trick as Unions type metadataUnion struct { dropbox.Tagged File *FileMetadata Folder *FolderMetadata } func IsMetadataFromJSON(data []byte) (IsMetadata, e) { var t metadataUnion if err := json.Unmarshal(data, &t); err != nil { return nil, err } switch t.Tag { ... } } #5: Inherited Types Problem: (de)serialization { ".tag": "file", "name": "Prime_Numbers.txt", "id": "id:a4ayc_80_OEAAAAAAAAAXw", "content_hash": "e3b0c44298fc" } OR { ".tag": "folder", "name": "math", "id": "id:a4ayc_80_OEAAAAAAAAAXz", "path_lower": "/homework/math", }
20. #6: Do NOT authenticate
21. #6: Do NOT authenticate • Apps authenticate, SDK accepts OAuth Token • Beware of known OAuth pitfalls ◦ Changed OAuth Endpoint from api.dropbox.com to api.dropboxapi.com ◦ App started failing with oauth2: cannot fetch token: 400 Bad Request ◦ Gem from oauth2/internal/token.go ◦ func RegisterBrokenAuthHeaderProvider(tokenURL string) ◦ Google, Stripe and yes, Dropbox are “broken” • More information: ◦ On debugging OAuth2 in #golang
22. #7: Auto Generate Tests
23. #7: Auto Generate Tests • Tests should be comprehensive and up-to-date • Dropbox SDK is NOT a good example! • Model after AWS SDK ◦ Example input/output defined in JSON ◦ Tests are auto-generated • Not a panacea, still need to write tests! ◦ Especially negative tests
24. #8: Handle errors the Go way
25. #8: Handle errors the Go way Dropbox SDK res, err := dbx.ListFolder(arg) if err != nil { switch e := err.(type) { case files.ListFolderAPIError: ... AWS SDK output, err := s3manage.Upload(...) if err != nil { if reqerr, ok := err.(RequestFailure); ok { ... • Have SDK errors implement the `error` interface • Use type assertions to extract errors • Good packages exist for combining / unrolling errors -- wish stdlib had more support!
26. Lessons Recap 1. Use a Code Generator 2. Avoid Surprises 3. Make Configuration Simple 4. Provide Visibility 5. Use Go Idioms for Unsupported Types 6. Do Not Authenticate 7. Auto Generate Tests 8. Handle Errors the Go Way
27. Where to get it? • SDK:https://github.com/dropbox/dropbox-sdk-go-unofficial • dbxcli: https://github.com/dropbox/dbxcli/ ◦ Command line tool for Dropbox ◦ Pre-compiled binaries for Windows, Mac, Linux, ARM ◦ Useful for end users as well as team administrators • Support for Dropbox Paper coming soon! ◦ https://www.dropbox.com/paper