Andrea Di Persio (SoundCloud), A tale of Go and Microservices, CodeFest 2017

https://2017.codefest.ru/lecture/1146

At SoundCloud we make extensive usage of Microservices, using many programming languages to solve a number of interesting problems.

With its lean footprint and batteries included standard library Go is a perfect choice for your next Microservice.

In this talk we will have a look at how we manage the deployment and monitoring of our Go Microservices and how it performs compared to other platforms.

Transcript

1. A tale of Go and Microservices Andrea Di Persio Backend Engineer
2. Agenda 1. Myself a. whoami b. sharing context c. tools of the trade
3. Agenda 2. The Go Programming Language a. The language b. Golang and Myself c. Golang and SoundCloud i. Infrastructure ii. Playback Team
4. Agenda 3. Microservices a. Definition b. Microservices and SoundCloud c. The Bounded Context d. Microservices and Go i. How we do it!
5. Myself
7. + =
8. + =
9. Working at SoundCloud since 2014. Daily work involve Go, Microservices, Cloud, etc. My team is responsible for media delivery and processing: ❏ Transcoding ❏ Streaming APIs and Protocols ❏ Streaming infrastructure (CDN, Storage, Cloud Computing, etc) ❏ Native players & Web players
10. Sharing Context
11. “What matters in writing software is not the look and feel of the experience, writing the code or the cultural implications of it.” Simple Made Easy by Ritch Hickey
12. What matters in software ❏ Does the software do what is supposed to do? ❏ Is it of high quality? ❏ Can we rely on it? ❏ Can problems be fixed along the way? ❏ Can requirements change over time? Simple Made Easy by Ritch Hickey
13. Tools of the trade ❏ Discussions ❏ Google Docs ❏ BSD/Linux terminal ❏ VIM, emacs editors ❏ git
14. The Go Programming Language
15. First appeared at the end of 2009. ❏ Maintained by Google ❏ Compiled, statically typed, automatic memory management through garbage collection ❏ CSP-style concurrent programming features ❏ Developed by some core UNIX contributors ❏ Leverage some technologies from Plan 9, Inferno operating systems
16. “Go is efficient, scalable, and productive. Some programmers find it fun to work in; others find it unimaginative, even boring. In this article we will explain why those are not contradictory positions.” Go at Google: Language Design in the Service of Software Engineering by Rob Pike
17. Go and Myself Go is extremely opinionated Less time is spent arguing and there are fewer good way of doing something. Code formatting is automatic, reading code is pleasant.
18. Go and Myself Go is batteries included The standard library does a fairly good job in providing all the building blocks required to build distributed systems. From reliable networking libraries to json encoders/decoders and testing.
19. Go and SoundCloud ❏ Infrastructure ❏ Bazooka - PaaS ❏ Kubernetes - PaaS ❏ Prometheus - Monitoring
20. Microservices
21. Trending since 2012. ❏ Many big adopters (Netflix, eBay, Spotify, etc). ❏ A different flavour of Service Oriented Architecture (SOA). ❏ Both addressing certain technological concern, but also offer new way of organizing work. Image from: Microservices Resource Guide by Martin Fowler
22. Are you ready for Microservices?
23. Can you... ❏ Understand Domain Driven Design? ❏ Deploy different stacks (JVM, Go, Ruby, etc)? ❏ Monitor many services? ❏ Live with Eventual consistency? ❏ Scale your network?
24. Bounded context The conditions under which a particular model is defined and applicable.
25. Bounded context Universal Track Model (Monolith) ❏ Name ❏ Creator ID ❏ Filename ❏ Genre ❏ Publication Date ❏ Artwork URL ❏ Format ❏ Bitrate ❏ Duration ❏ Description ❏ Label ID ❏ Label Name ❏ Is Public? ❏ Release Year ❏ Release Month ❏ ... +60 Fields!!!
26. Bounded context Streaming Track Model (Microservice) ❏ Creator ID ❏ Filename ❏ Duration ❏ Format ❏ Bitrate
27. SoundCloud and Microservices
28. Illustration by Sandra Brandstätter
31. Backend For Frontend (BFF)
32. “As we're now recognizing, often the best way to find out if a software idea is useful is to build a simplistic version of it and see how well it works out. During this first phase you need to prioritize speed (and thus cycle time for feedback), so the premium of microservices is a drag you should do without.” Martin Fowler MonolithFirst
33. MicroservicePremium by Martin Fowler
34. “Microservices are not something you should aim for. They are a means to an end.” “Focus on what's important - building useful software.” Sam Newman
35. Microservices and Myself What I like
36. Confidence ❏ Dependencies are less, and better understood. ❏ Encourage experimenting. ❏ Promote ownership.
37. Isolation ❏ Reduced metrics noise. ❏ Optimise for specific performance requirements. ❏ Isolate failures.
38. Developer friendly ❏ The Bounded context helps reasoning without being overwhelming. ❏ New recruit can be onboarded on specific features and gradually get familiar with the team owned sub systems.
39. Microservices and Myself What I don’t like
40. Autonomy ❏ It emphasize the role a Lead. ❏ Is an hard job to onboard every team member on Domain Driven Design. ❏ It can promote isolation and disconnection from the product.
41. Confusion ❏ Who owns what? ❏ Building an Ubiquitous Language should be a constant effort, “but we don’t have time to write documentation”.
42. Decentralization ❏ Finding ‘things’ rely on pragmatic naming practice (what the banana service is supposed to do?). ❏ Dependency management.
43. Microservices and Go How we do it!
44. Project Layout ❏ A repository contains everything required to deploy a single microservice ❏ Source code ❏ Deployment configuration ❏ Dependencies are frozen in the vendor folder.
49. manifest.json
50. Makefile targets ❏ assemble: Collect all files required to build an artifact. ❏ package: Create the artifact for the specified platform. ❏ publish: Upload to the test registry. ❏ publish-deploy: Prepare the deployment ❏ Configure routing (service discovery, lb, etc) ❏ Configure component
51. Makefile targets ❏ deploy: perform the deploy of a component, the specific version to the specified number of replicas. ❏ promote: If the deploy have succeeded we promote the package to stable meaning other systems can integrate with this version.
52. Testing ❏ The Integration test is a regular parametrized Go Test which is compiled and bundled in the artifact. ❏ During the deploy phase we launch a control instance which is not receiving any traffic. ❏ The integration test is run against this instance. ❏ The deploy is aborted if the integration test fails.
53. Monitoring
54. Monitoring var errorCounter = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "fetch_errors_total", Help: "Count of fetch errors by upstream.", }, []string{"upstream”}, ) func init() { prometheus.MustRegister(errorCounter) }
55. Monitoring func fetch() { resp, err := http.Get(“http://storage-server.net”) if err != nil { errorCounter.WithLabelValues(“storage-server”).Inc() ... } ... }
56. Monitoring func main() { … http.Handle(“/metrics”, prometheus.Handler()) }
57. Monitoring # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0.000351319 go_gc_duration_seconds{quantile="0.25"} 0.00043683700000000003 go_gc_duration_seconds{quantile="0.5"} 0.000526461 go_gc_duration_seconds{quantile="0.75"} 0.0006740310000000001 go_gc_duration_seconds{quantile="1"} 0.008426264000000001 go_gc_duration_seconds_sum 8.51508857 go_gc_duration_seconds_count 9893 # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 21
59. Monitoring
61. Microservices and Go What works well
62. Compiled ❏ Reduced size of artifacts. ❏ Fast startup time. ❏ Faster deploys.
63. Vendoring ❏ Freezing the dependency graph using vendoring for no hassle reproducible builds. ❏ No need for a central repository.
64. Libraries ❏ HTTP and JSON are the bread and butter of microservices, the standard library provide both an excellent HTTP server and JSON encoding libraries. ❏ Many well written and maintained libraries for interop with common technologies (MySQL, RabbitMQ, AWS, Google, Prometheus, etc).
65. Toolchain ❏ Handle automatic code formatting! ❏ Race detection. ❏ pprof exporter. ❏ Runtime inspection. ❏ Cross Compilation support.
66. Microservices and Go What does not work that well
67. Vendoring ❏ Freezing the dependency graph using vendoring. ❏ Versioning? ❏ go get?
68. C interop ❏ Increase builds complexity. ❏ Cannot use cross compilation. ❏ Where are my tools? ❏ Cannot use C specific tools (valgrind). ❏ Go toolchain effectiveness is reduced. ❏ ‘CGO is not Go’.
69. SoundCloud Backstage Blog
70. Illustration by Sandra Brandstätter
71. Backend Engineer @andreadipersio Andrea Di Persio SoundCloud Questions? andreadipersio www.andreadipersio.com andrea@soundcl oud.com