Migrate to a New Platform by Creating a Pipeline in Go (Julia Wong 2019)

Transcript

1. JULIA WONG | DEVELOPER @ ATLASSIAN Migrate to a new platform by creating a pipeline in Go
2. Agenda Previous Pipeline Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary
3. VENDOR SERVICE STATSD SERVER
4. Trial vendors Multiple vendors and it can’t disrupt our users Send in parallel Can’t afford a big bang cut-over. Need time to migrate from old to new vendor. Visibility What metrics are being sent? How much? Pipeline Needs to solve
5. Trial vendors Multiple vendors and it can’t disrupt our users Send in parallel Can’t afford a big bang cut-over. Need time to migrate from old to new vendor. Visibility What metrics are being sent? How much? Pipeline Needs to solve
6. Trial vendors Multiple vendors and it can’t disrupt our users Send in parallel Can’t afford a big bang cut-over. Need time to migrate from old to new vendor. Visibility What metrics are being sent? How much? Pipeline Needs to solve
7. VENDOR SERVICE STATSD SERVER
8. Agenda Previous Pipeline Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary
9. OLD VENDOR SERVICE STATSD SERVER
10. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY
11. return &httputil.ReverseProxy { FlushInterval: 5, // Modifies the request Director: director, } Reverse Proxy Proxy Metrics Compression
12. return &httputil.ReverseProxy { FlushInterval: 5, // Modifies the request Director: director, } func director() func(req *http.Request) { return func(req *http.Request) { // Emit metrics about payload } } Reverse Proxy Proxy Metrics Compression
13. return &httputil.ReverseProxy { FlushInterval: 5, // Modifies the request Director: director, } func director() func(req *http.Request) { return func(req *http.Request) { // Emit metrics about payload // Compress payload } } Reverse Proxy Proxy Metrics Compression
14. 1Mb 65Kb Compression 94% smaller!
15. Metrics 10Mb/sec, 600,000 metrics/sec 800GB/day, 50,000 million metrics/day
16. Agenda Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary Previous Pipeline
17. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY
18. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY KINESIS TRIAL VENDORS STREAM WORKERS
19. Stream composed of shards Shards can split and merge Stream workers need to determine which shards to process Lease - associate worker with shard/s KINESIS
20. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY KINESIS TRIAL VENDORS STREAM WORKERS
21. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY KINESIS NEW VENDOR STREAM WORKERS
22. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY KINESIS NEW VENDOR STREAM WORKERS
23. Agenda Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary Previous Pipeline
24. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY KINESIS NEW VENDOR STREAM WORKERS
25. type OldDatapoint struct { Points []*[2]float64 `json:"points"` Metric string `json:"metric"` Type string `json:"type"` Interval float64 `json:"interval"` } var point OldDatapoint err := json.Unmarshal(data, &point) Metric Proxy JSON Conversion
26. type OldDatapoint struct { Points []*[2]float64 `json:"points"` Metric string `json:"metric"` Type string `json:"type"` Interval float64 `json:"interval"` } func translate (o *OldDatapoint) { datapoint := &newVendor.Datapoint { Metric: o.Metric, Value: int64(o.Points[0][1]), Time: time.Unix(int64(o.Point[0][0]), 0) } } Metric Proxy JSON Conversion
27. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS
28. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS
29. Agenda Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary Previous Pipeline
30. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS
31. OLD VENDOR SERVICE STATSD SERVER METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS github.com/atlassian/gostatsd
32. Forwarder and Aggregator STATSD
 (AGGREGATOR) SERVICE STATSD
 (FORWARDER) UDP
 RAW METRICS JSON
 AGGREGATED METRICS
 PROTOBUF
 CONSOLIDATED METRICS
33. OLD VENDOR SERVICE STATSD SERVER V1 METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS
34. OLD VENDOR SERVICE STATSD SERVER V1 SERVICE + STATSD SIDECAR V2 STATSD SERVER V2 METRICS PROXY NEW VENDOR KINESIS STREAM WORKERS
35. Benchmark and Expvar
36. Agenda Reverse Proxy Kinesis Metrics Proxy Gostatsd Summary Previous Pipeline
37. VENDOR SERVICE STATSD SERVER V1
38. STATSD SERVER V2 SERVICE + STATSD SIDECAR V2 NEW VENDOR METRICS PROXY KINESIS STREAM WORKERS
39. Benchmarking and expvar Minimal work to use tooling Kinesis stream worker Go workers aren’t on par with Java workers, yet httputil package Creating a reverse proxy was easy! Learnings Go
40. Benchmarking and expvar Minimal work to use tooling Learnings Go Kinesis stream worker Go workers aren’t on par with Java workers, yet httputil package Creating a reverse proxy was easy!
41. Benchmarking and expvar Minimal work to use tooling Learnings Go Kinesis stream worker Go workers aren’t on par with Java workers, yet httputil package Creating a reverse proxy was easy!
42. Insights and control Knowledge is power Experiment safely Have a way to test changes by forking the data Add an interface early Put something between you and your vendor Learnings Pipeline
43. Insights and control Knowledge is power Experiment safely Have a way to test changes by forking the data Add an interface early Put something between you and your vendor Learnings Pipeline
44. Insights and control Knowledge is power Experiment safely Have a way to test changes by forking the data Add an interface early Put something between you and your vendor Learnings Pipeline
45. Thank you! JULIA WONG | DEVELOPER @ ATLASSIAN