From Service to Platform: A Ranking System in Go

What started out as an experimental service to rank livestreams evolved to a platform powering all types of content recommendations. Go's stringent code philosophy paved the way to a modular pipeline-based system for scatter-gather workflows enabling anyone to add new ranking algorithms.

GopherCon Europe 2022, Berlin
https://www.youtube.com/watch?v=5jSyctW1rPg

GopherCon UK 2022, London
https://www.youtube.com/watch?v=TNyoKBLxfTM

Transcript

1. From Service To Platform A Ranking System in Go Konrad Reiche r/streetwear r/aww r/dogecoin r/golang r/yoga
2. Once upon a time…
3. Once upon a time… … or more like last year
4. Once upon a time… … or more like last year Redis Cluster
5. Once upon a time… … or more like last year Redis Cluster
6. Once upon a time… … or more like last year Redis Cluster
7. Redis Monitor 1655767305.672853 [0 10.8.144.28:61122] "get" "post:3hg9w" 1655767305.672862 [0 10.8.144.28:61122] "get" "post:5cm6n9" 1655767305.672869 [0 10.8.144.28:61122] "get" "post:62s7bk" 1655767305.672876 [0 10.8.144.28:61122] "get" "post:2zqpf" 1655767305.672880 [0 10.8.144.28:61122] "get" "post:2fn65o" ...
8. Redis Monitor 1655767305.672853 [0 10.8.144.28:61122] "get" "post:3hg9w" 1655767305.672862 [0 10.8.144.28:61122] "get" "post:5cm6n9" 1655767305.672869 [0 10.8.144.28:61122] "get" "post:62s7bk" 1655767305.672876 [0 10.8.144.28:61122] "get" "post:2zqpf" 1655767305.672880 [0 10.8.144.28:61122] "get" "post:2fn65o" ... Timestamp Database, Network Address Command and Key
9. Redis Monitor 1655767305.672853 [0 10.8.144.28:61122] "get" "post:3hg9w" 1655767305.672862 [0 10.8.144.28:61122] "get" "post:5cm6n9" 1655767305.672869 [0 10.8.144.28:61122] "get" "post:62s7bk" 1655767305.672876 [0 10.8.144.28:61122] "get" "post:2zqpf" 1655767305.672880 [0 10.8.144.28:61122] "get" "post:2fn65o" ...
10. Redis Monitor 1655767305.672853 [0 10.8.144.28:61122] "get" "deleted_posts" 1655767305.672862 [0 10.8.144.28:61122] "get" "post:5cm6n9" 1655767305.672869 [0 10.8.144.28:61122] "get" "deleted_posts" 1655767305.672876 [0 10.8.144.28:61122] "get" "deleted_posts" 1655767305.672880 [0 10.8.144.28:61122] "get" "post:2fn65o" ...
11. func main() { flag.Parse() b, err := os.ReadFile(path) if err != nil { log.Fatal(err) } countByKey := make(map[string]int) lines := strings.Split(string(b), "\n") for _, line := range lines { split := strings.Split(line, " ") if len(split) < 5 { continue } key := split[4] countByKey[key] += 1 } type keyCount struct { key string count int } counts := make([]keyCount, 0, len(countByKey)) for key, count := range countByKey { counts = append(counts, keyCount{key: key, count: count}) } sort.Slice(counts, func(i, j int) bool { return counts[i].count > counts[j].count }) for i := 0; i < topKeys; i++ { fmt.Println(keyCounts[i].count, keyCounts[i].key) } }
12. func main() { flag.Parse() b, err := os.ReadFile(path) if err != nil { log.Fatal(err) } countByKey := make(map[string]int) lines := strings.Split(string(b), "\n") for _, line := range lines { split := strings.Split(line, " ") if len(split) < 5 { continue } key := split[4] countByKey[key] += 1 } type keyCount struct { key string count int } counts := make([]keyCount, 0, len(countByKey)) for key, count := range countByKey { counts = append(counts, keyCount{key: key, count: count}) } sort.Slice(counts, func(i, j int) bool { return counts[i].count > counts[j].count }) for i := 0; i < topKeys; i++ { fmt.Println(keyCounts[i].count, keyCounts[i].key) } } Parse flags, read file into memory
13. func main() { flag.Parse() b, err := os.ReadFile(path) if err != nil { log.Fatal(err) } countByKey := make(map[string]int) lines := strings.Split(string(b), "\n") for _, line := range lines { split := strings.Split(line, " ") if len(split) < 5 { continue } key := split[4] countByKey[key] += 1 } type keyCount struct { key string count int } counts := make([]keyCount, 0, len(countByKey)) for key, count := range countByKey { counts = append(counts, keyCount{key: key, count: count}) } sort.Slice(counts, func(i, j int) bool { return counts[i].count > counts[j].count }) for i := 0; i < topKeys; i++ { fmt.Println(keyCounts[i].count, keyCounts[i].key) } } Parse flags, read file into memory Parse each line, split by column and count keys
14. func main() { flag.Parse() b, err := os.ReadFile(path) if err != nil { log.Fatal(err) } countByKey := make(map[string]int) lines := strings.Split(string(b), "\n") for _, line := range lines { split := strings.Split(line, " ") if len(split) < 5 { continue } key := split[4] countByKey[key] += 1 } type keyCount struct { key string count int } counts := make([]keyCount, 0, len(countByKey)) for key, count := range countByKey { counts = append(counts, keyCount{key: key, count: count}) } sort.Slice(counts, func(i, j int) bool { return counts[i].count > counts[j].count }) for i := 0; i < topKeys; i++ { fmt.Println(keyCounts[i].count, keyCounts[i].key) } } Parse flags, read file into memory Parse each line, split by column and count keys Convert to slice of tuples, sort and print
15. Redis: Finding Hot Keys $ ./find-hotkeys -file monitor.log -n 5
16. Redis: Finding Hot Keys $ ./find-hotkeys -file monitor.log -n 5 8252 "deleted_posts" 1907 "post:2tk95" 1756 "post:2xcv7" 772 "post:3nasz" 509 "post:2qjpg"
17. $ cat monitor.log | awk '{print $4}' | sort | uniq -c | sort -nr | head -n 5
18. UNIX Pipes Output log Print 4th column Count unique lines Sort numerical Print first 5 lines The same can be achieved with existing programs and UNIX pipes. $ cat monitor.log | awk '{print $4}' | sort | uniq -c | sort -nr | head -n 5
19. UNIX Pipes Output log Print 4th column Count unique lines Sort numerical Print first 5 lines The same can be achieved with existing programs and UNIX pipes. UNIX Toolbox Philosophy Write programs that: • Do one thing well • Compose • Easily communicate $ cat monitor.log | awk '{print $4}' | sort | uniq -c | sort -nr | head -n 5
20. 20 Essential Complexity
21. 21 Essential Complexity Accidental Complexity
22. 22 Essential Complexity Accidental Complexity No Silver Bullet—Essence and Accident in Software Engineering Brooks, Frederick P. (1986)
23. 23 Go is a language that helps us to reduce accidental complexity
24. Konrad Reiche Ranking Platform, Reddit
25. Konrad Reiche Ranking Platform, Reddit
26. Konrad Reiche Ranking Platform, Reddit
27. What is a ranking (recommendation) system? A recommendation system helps users to find content they find compelling.
28. What is a ranking (recommendation) system? A recommendation system helps users to find content they find compelling. 1. Candidate Generation Start from a potentially huge corpus and generate a much smaller subset of candidates.
29. What is a ranking (recommendation) system? A recommendation system helps users to find content they find compelling. 1. Candidate Generation Start from a potentially huge corpus and generate a much smaller subset of candidates. 2. Filtering Some candidates should be removed, for example content already watched or content the user marked as something they do not want to consume.
30. What is a ranking (recommendation) system? A recommendation system helps users to find content they find compelling. 1. Candidate Generation Start from a potentially huge corpus and generate a much smaller subset of candidates. 2. Filtering Some candidates should be removed, for example content already watched or content the user marked as something they do not want to consume. 3. Scoring Assign scores to sort the candidates.
31. 31 From Service to Platform: A Ranking System in Go Example: Ranking Service
32. 32 From Service to Platform: A Ranking System in Go Popular Posts Example: Ranking Service
33. 33 From Service to Platform: A Ranking System in Go Popular Posts Fetch posts Example: Ranking Service
34. 34 From Service to Platform: A Ranking System in Go Popular Posts Fetch posts Filter posts Example: Ranking Service
35. 35 From Service to Platform: A Ranking System in Go Popular Posts Fetch posts Filter posts User Post Views Example: Ranking Service
36. 36 From Service to Platform: A Ranking System in Go Example: Ranking Service Popular Posts Fetch posts Filter posts Score posts User Post Views
37. 37 From Service to Platform: A Ranking System in Go Example: Ranking Service Popular Posts Fetch posts Filter posts Score posts User Post Views Model
38. 38 From Service to Platform: A Ranking System in Go Example: Ranking Service Popular Posts Fetch posts Filter posts Score posts User Post Views Model
39. 39 From Service to Platform: A Ranking System in Go Example: Ranking Service Popular Posts Fetch posts Filter posts Score posts Video Posts User Post Views Model
40. 40 From Service to Platform: A Ranking System in Go Example: Ranking Service Popular Posts Fetch posts Filter posts Score posts Video Posts User Post Views Model remove duplicates
41. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
42. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
43. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
44. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
45. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
46. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
47. 47 We continuously refactor to reduce the accidental complexity of code
48. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
49. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
50. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
51. Example: Ranking Service func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) { posts, err := s.fetchPopularAndVideoPosts(ctx) if err != nil { return nil, err } imagePosts, err := s.cache.FetchImagePosts(ctx) if err != nil { return nil, err } posts = s.filterPosts(posts, imagePosts) posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts) if err != nil { return nil, err } posts = s.sortPosts(posts, scores) return pb.NewPopularFeed(posts), nil }
52. 52 Can refactoring be limited through a structural design?
53. 53 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Candidate Generation Filter Score
54. 54 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 …
55. 55 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) }
56. 56 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) }
57. 57 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) } type Request struct { Context *Entity Candidates []*Entity }
58. 58 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) } type Request struct { Context *Entity Candidates []*Entity }
59. 59 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity }
60. 60 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity }
61. 61 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity } type Entity struct { ID string Features map[string]*Feature Score float64 }
62. 62 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity } type Entity struct { ID string Features map[string]*Feature Score float64 }
63. 63 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity } type Entity struct { ID string Features map[string]*Feature Score float64 }
64. 64 From Service to Platform: A Ranking System in Go type Request struct { Context *Entity Candidates []*Entity } type Entity struct { ID string Features map[string]*Feature Score float64 }
65. 65 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) } type Request struct { Context *Entity Candidates []*Entity }
66. 66 From Service to Platform: A Ranking System in Go UNIX Toolbox Philosophy Stage 1 Stage 2 … type Stage interface { Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) } type Request struct { Context *Entity Candidates []*Entity }
67. gRPC Protobuf Definition syntax = "proto3"; service Ranking { rpc Rank (Request) returns (Request); }
68. gRPC Protobuf Definition syntax = "proto3"; service Ranking { rpc Rank (Request) returns (Request); } message Request { Entity context = 1; repeated Entity candidates = 2; RequestOptions options = 3; }
69. gRPC Protobuf Definition message Entity { string id = 1; map<string, Feature> features = 2; double score = 3; } message Feature { oneof value { string as_string = 1; int64 as_int = 2; double as_float = 3; bool as_bool = 4; // ... }; }
70. gRPC Protobuf Request Example context: { id: "t2_bd5ts" features: { key: "geo_city" value: { as_string: "SAN_FRANCISCO" } } features: { key: "geo_country" value: { as_string: "US" } } } options: { method: "rank_popular_feed" limit: 20 }
71. gRPC Service type server struct { *grpc.Server stage stage.Stage } func (s *server) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { return s.stage.Rank(ctx, req) }
72. 72 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit
73. 73 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Fetch Image Posts
74. 74 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Fetch Image Posts Series
75. 75 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Fetch Image Posts Series
76. 76 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Fetch Image Posts Series
77. 77 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Filter Recently Viewed Posts
78. 78 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates
79. 79 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
80. 80 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
81. Stage: Fetch Popular Posts type fetchPopularPosts struct { cache *store.PostCache } func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts { return &fetchPopularPosts{cache: cache} } func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { postIDs, err := s.cache.FetchPopularPostIDs(ctx) if err != nil { return nil, err } for _, id := range postIDs { req.Candidates = append(req.Candidates, pb.NewCandidate(postID)) } return req, nil }
82. Stage: Fetch Popular Posts type fetchPopularPosts struct { cache *store.PostCache } func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts { return &fetchPopularPosts{cache: cache} } func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { postIDs, err := s.cache.FetchPopularPostIDs(ctx) if err != nil { return nil, err } for _, id := range postIDs { req.Candidates = append(req.Candidates, pb.NewCandidate(id)) } return req, nil }
83. Stage: Fetch Popular Posts type fetchPopularPosts struct { cache *store.PostCache } func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts { return &fetchPopularPosts{cache: cache} } func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { postIDs, err := s.cache.FetchPopularPostIDs(ctx) if err != nil { return nil, err } for _, id := range postIDs { req.Candidates = append(req.Candidates, pb.NewCandidate(id)) } return req, nil }
84. Stage: Fetch Popular Posts type fetchPopularPosts struct { cache *store.PostCache } func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts { return &fetchPopularPosts{cache: cache} } func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { postIDs, err := s.cache.FetchPopularPostIDs(ctx) if err != nil { return nil, err } for _, id := range postIDs { req.Candidates = append(req.Candidates, pb.NewCandidate(id)) } return req, nil }
85. Stage: Fetch Popular Posts type fetchPopularPosts struct { cache *store.PostCache } func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts { return &fetchPopularPosts{cache: cache} } func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { postIDs, err := s.cache.FetchPopularPostIDs(ctx) if err != nil { return nil, err } for _, id := range postIDs { req.Candidates = append(req.Candidates, pb.NewCandidate(id)) } return req, nil }
86. 86 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
87. Stage: Filtering Recently Viewed Posts func (s *filterRecentlyViewedPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { seen := req.Context.Features["recently_viewed_post_ids"].GetAsBoolMap() var filtered []*pb.Entity for _, candidate := range req.Candidates { if !seen[candidate.Id] { filtered = append(filtered, candidate) } } req.Candidates = filtered return req, nil }
88. Stage: Filtering Recently Viewed Posts func (s *filterRecentlyViewedPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { seen := req.Context.Features["recently_viewed_post_ids"].GetAsBoolMap() var filtered []*pb.Entity for _, candidate := range req.Candidates { if !seen[candidate.Id] { filtered = append(filtered, candidate) } } req.Candidates = filtered return req, nil }
89. Stage: Filtering Recently Viewed Posts func (s *filterRecentlyViewedPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { seen := req.Context.Features["recently_viewed_post_ids"].GetAsBoolMap() n := 0 for _, candidate := range req.Candidates { if !seen[candidate.Id] { req.Candidates[n] = candidate n++ } } req.Candidates = req.Candidates[:n] // in-place filtering return req, nil }
90. 90 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
91. 91 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
92. Meta-Stage: Series type series struct { stages []Stage } func Series(stages ...Stage) *series { return &series{stages: stages} } func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { var err error resp := req for _, stage := range s.stages { resp, err = stage.Rank(ctx, req) if err != nil { return nil, err } req = resp } return resp, nil }
93. type series struct { stages []Stage } func Series(stages ...Stage) *series { return &series{stages: stages} } func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { var err error resp := req for _, stage := range s.stages { resp, err = stage.Rank(ctx, req) if err != nil { return nil, err } req = resp } return resp, nil } Meta-Stage: Series
94. type series struct { stages []Stage } func Series(stages ...Stage) *series { return &series{stages: stages} } func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { var err error resp := req for _, stage := range s.stages { resp, err = stage.Rank(ctx, req) if err != nil { return nil, err } req = resp } return resp, nil } Meta-Stage: Series
95. type series struct { stages []Stage } func Series(stages ...Stage) *series { return &series{stages: stages} } func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { var err error resp := req for _, stage := range s.stages { resp, err = stage.Rank(ctx, req) if err != nil { return nil, err } req = resp } return resp, nil } Meta-Stage: Series
96. type series struct { stages []Stage } func Series(stages ...Stage) *series { return &series{stages: stages} } func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { var err error resp := req for _, stage := range s.stages { resp, err = stage.Rank(ctx, req) if err != nil { return nil, err } req = resp } return resp, nil } Meta-Stage: Series
97. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) }
98. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) } golang.org/x/sync/errgroup
99. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) } Calls the function in a goroutine, first non-nil error to be returned cancels the group
100. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) }
101. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) }
102. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) }
103. Meta-Stage: If-Else type Selector func(context.Context, *pb.Request) bool func (s *ifElse) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { if s.selector(ctx, req) { return s.ifStage.Rank(ctx, req) } return s.elseStage.Rank(ctx, req) }
104. 104 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Fetch Popular Posts Fetch Video Posts Series Series Parallel Merge Candidates Fetch Recently Viewed Posts Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit Series Fetch Image Posts Series Score Candidates Filter Recently Viewed Posts Sort Candidates Candidates Features Filtering Meta-Stages
105. 105 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit func PopularFeed(d *service.Dependencies) stage.Stage { return stage.Series( stage.Parallel(merger.MergeCandidates, stage.FetchPopularPosts(d.PostCache), stage.FetchVideoPosts(d.PostCache), stage.FetchImagePosts(d.PostCache), ), stage.FetchRecentlyViewedPosts(d.UserPostViews), stage.FilterRecentlyViewedPosts(), stage.ScoreCandidates(d.RankingModel), stage.SortCandidates(), ) }
106. From Service to Platform When does a service become a platform?
107. From Service to Platform When does a service become a platform? Service Customers are the users / product management
108. From Service to Platform When does a service become a platform? Service Customers are the users / product management Platform Customers are engineers developing on the service
109. 109 From Service to Platform: A Ranking System in Go What is your API? Path /api/user/{id}.json Description Get a user object Method GET
110. package user type User struct { ID string Name string permissions []string } 110 From Service to Platform: A Ranking System in Go What is your API?
111. package user type User struct { ID string Name string permissions []string } 111 From Service to Platform: A Ranking System in Go What is your API?
112. package user type User struct { ID string Name string permissions []string } 112 From Service to Platform: A Ranking System in Go What is your API?
113. package user type User struct { ID string Name string permissions []string } 113 From Service to Platform: A Ranking System in Go What is your API?
114. package user type User struct { ID string Name string permissions []string } 114 From Service to Platform: A Ranking System in Go What is your API?
115. package user type User struct { ID string Name string permissions []string } 115 From Service to Platform: A Ranking System in Go What is your API?
116. package user type User struct { ID string Name string permissions []string } 116 From Service to Platform: A Ranking System in Go What is your API? • Start with unexported identifiers • Make decisions for exporting identifiers based on how you want them to be consumed
117. 117 A General-Purpose Ranking Service From Service to Platform: A Ranking System in Go Quickly and flexibly perform complex scatter-gather ranking workflows at Reddit func PopularFeed(d *service.Dependencies) stage.Stage { return stage.Series( stage.Parallel(merger.MergeCandidates, stage.FetchPopularPosts(d.PostCache), stage.FetchVideoPosts(d.PostCache), stage.FetchImagePosts(d.PostCache), ), stage.FetchRecentlyViewedPosts(d.UserPostViews), stage.FilterRecentlyViewedPosts(), stage.ScoreCandidates(d.RankingModel), stage.SortCandidates(), ) }
118. 118 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service
119. 119 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed
120. 120 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed …
121. 121 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed Popular Feed …
122. 122 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed Home Feed Popular Feed …
123. 123 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed Home Feed Popular Feed … …
124. 124 Transition to Platform From Service to Platform: A Ranking System in Go Ranking Service Livestream Feed Home Feed Popular Feed … Ranking Platform …
125. package stage 125 Maintaining the Platform From Service to Platform: A Ranking System in Go Series Parallel Shuffle Candidates Fetch Posts Filter By Feature … If-Else Fetch Subscriptions
126. package stage 126 Maintaining the Platform From Service to Platform: A Ranking System in Go Series Parallel Shuffle Candidates Fetch Posts Filter By Feature … If-Else Fetch Subscriptions Filter Private Posts Filter by Timestamp Shuffle Topics …
127. Creating A Toolbox Four Principles for a Platform API 127 From Service to Platform: A Ranking System in Go Limited Scope Clear Naming Decoupling Strive for Reuse
128. Applying the Four Principles const shuffleProbability = 0.2 type imagePosts struct { cache *store.PostCache } func (s *imagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil }
129. Applying the Four Principles const shuffleProbability = 0.2 type imagePosts struct { cache *store.PostCache } func (s *imagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil }
130. Applying the Four Principles const shuffleProbability = 0.2 type imagePosts struct { cache *store.PostCache } func (s *imagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil }
131. Applying the Four Principles const shuffleProbability = 0.2 type imagePosts struct { cache *store.PostCache } func (s *imagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Clear Naming
132. Applying the Four Principles const shuffleProbability = 0.2 type fetchImagePosts struct { cache *store.PostCache } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Clear Naming
133. Applying the Four Principles const shuffleProbability = 0.2 type fetchImagePosts struct { cache *store.PostCache } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } s.shufflePostIDs(postIDs, shuffleProbability) for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Limited Scope
134. Applying the Four Principles type fetchImagePosts struct { cache *store.PostCache } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Limited Scope
135. Applying the Four Principles type fetchImagePosts struct { cache *store.PostCache } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Strive for Reuse
136. Applying the Four Principles type fetchImagePosts struct { cache *store.PostCache } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature("subreddit_ids") postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Strive for Reuse
137. Applying the Four Principles type fetchImagePosts struct { cache *store.PostCache subredditFeature string } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature(s.subredditFeature) postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs) if err != nil { return nil, err } for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Decoupling
138. Applying the Four Principles type fetchImagePosts struct { cache *store.PostCache subredditFeature string } func (s *fetchImagePosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { subredditIDs := req.Context.GetStringArrayFeature(s.subredditFeature) postIDs, err := s.cache.FetchImagePosts(ctx, subredditIDs...) if err != nil { return nil, err } for _, postID := range postIDs { req.Candidates = append(req.Candidates, pb.Candidate(postID)) } return req, nil } Decoupling
139. 139 Reuse Clarity
140. 140 Single Method Interface From Service to Platform: A Ranking System in Go
141. “The bigger the interface, the weaker the abstraction” ― Rob Pike
142. Functions Implementing Interfaces “The bigger the interface, the weaker the abstraction” ― Rob Pike type RankFunc func(context.Context, *pb.Request) (*pb.Request, error)
143. Functions Implementing Interfaces “The bigger the interface, the weaker the abstraction” ― Rob Pike type RankFunc func(context.Context, *pb.Request) (*pb.Request, error) func (f RankFunc) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { return f(ctx, req) }
144. Functions Implementing Interfaces “The bigger the interface, the weaker the abstraction” ― Rob Pike type RankFunc func(context.Context, *pb.Request) (*pb.Request, error) func (f RankFunc) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { return f(ctx, req) }
145. func Pipeline(d *service.Dependencies) stage.Stage { return stage.Series( stage.FetchSubscriptions(d.SubscriptionService), stage.FetchPosts(d.Cache), stage.FilterPostsForUK(), stage.ShufflePosts(0.2), ) } Anonymous Interface Implementations
146. func Pipeline(d *service.Dependencies) stage.Stage { return stage.Series( stage.FetchSubscriptions(d.SubscriptionService), stage.FetchPosts(d.Cache), stage.FilterPostsForUK(), stage.ShufflePosts(0.2), ) } Anonymous Interface Implementations
147. func Pipeline(d *service.Dependencies) stage.Stage { return stage.Series( stage.FetchSubscriptions(d.SubscriptionService), stage.FetchPosts(d.Cache), stage.FilterPostsForUK(), stage.ShufflePosts(0.2), ) } Anonymous Interface Implementations
148. func Pipeline(d *service.Dependencies) stage.Stage { return stage.Series( stage.FetchSubscriptions(d.SubscriptionService), stage.FetchPosts(d.Cache), stage.RankFunc(func(context.Context, *pb.Request) (*pb.Request, error) { if req.Context.Features["geo_country"] == "uk" { // ... } return req, nil }), stage.ShufflePosts(0.2), ) } Anonymous Interface Implementations
149. Middlewares
150. Middlewares type Middleware func(stage Stage) Stage
151. Middlewares type Middleware func(stage Stage) Stage func ExampleMiddleware(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) { // ... return next.Rank(ctx, req) }) }
152. func Monitor(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) { startedAt := time.Now() method := req.Options.Method defer func() { stageLatencySeconds.With(prometheus.Labels{ methodLabel: method, stageLabel: stage.Name(next), }).Observe(time.Since(startedAt).Seconds()) }() return next.Rank(ctx, req) }) } Middleware: Monitor
153. func Monitor(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) { startedAt := time.Now() method := req.Options.Method defer func() { stageLatencySeconds.With(prometheus.Labels{ methodLabel: method, stageLabel: stage.Name(next), }).Observe(time.Since(startedAt).Seconds()) }() return next.Rank(ctx, req) }) } Middleware: Monitor Record elapsed time in deferred statement Delegate to underlying stage
154. Middleware: Monitor makes pprof obsolete
155. Middleware: Monitor makes pprof obsolete
156. Middleware: Log func Log(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (resp *pb.Request, err error) { defer func() { if err != nil { log.Errorw( "stage failed", "error", err, "request", req.JSON(), "response", resp.JSON(), "stage", stage.Name(stage), ) } }() return stage.Rank(ctx, req) }) }
157. Middleware: Log func Log(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (resp *pb.Request, err error) { defer func() { if err != nil { log.Errorw( "stage failed", "error", err, "request", req.JSON(), "response", resp.JSON(), "stage", stage.Name(stage), ) } }() return stage.Rank(ctx, req) }) } func (r *Request) JSON() string { b, err := protojson.Marshal(r) if err != nil { log.Error(err) return "" } return string(b) }
158. Middleware: Feature Flags for Incident Mitigation func FeatureFlag(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) { key := "feature_flag.stage." + stage.Name(current) if !liveconfig.GetBool(key) { return req, nil } return next.Rank(ctx, req) }) }
159. Middleware: Feature Flags for Incident Mitigation func FeatureFlag(next stage.Stage) stage.Stage { return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) { key := "feature_flag.stage." + stage.Name(current) if !liveconfig.GetBool(key) { return req, nil } return next.Rank(ctx, req) }) } Skip stage entirely
160. Accidental Complexity
161. It’s Go time… Accidental Complexity
162. A Framework for Refactoring • Being forced to write your business logic into small components helped to limit accidental complexity but it did not eliminate the need for refactoring • Our ability to refactor increased due stage interface providing a framework • This framework requires a set of principles for designing those components • Any code contributed to middlewares or meta-stages pays off due to their multiplier effect
163. • Platform-centric thinking starts with the first developers outside of our team contributing code—this is not something you can simulate. • Providing an opinionated framework will create friction. • Ways to resolve confusion or disagreement: 1. Enforce the existing design 2. Quick-and-dirty workaround 3. Rethink the existing design Platform Building is a Discourse
164. 164 What’s next?
165. 165 What’s next? Error Semantics
166. Meta-Stage: Parallel func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) { resps := make([]*pb.Request, len(s.stages)) g, groupCtx := errgroup.WithContext(ctx) for i := range s.stages { i := i g.Go(func() error { defer log.CapturePanic(groupCtx) resp, err := s.stages[i].Rank(groupCtx, pb.Copy(req)) if err != nil { return err } resps[i] = resp return nil }) } if err := g.Wait(); err != nil { return nil, err } return s.merge(ctx, req, resps...) }
167. 167 What’s next? Error Semantics
168. 168 What’s next? Error Semantics stage.Parallel(merger.MergeCandidates, stage.FetchPopularPosts(d.PostCache), stage.FetchVideoPosts(d.PostCache), stage.FetchImagePosts(d.PostCache), )
169. 169 What’s next? Error Semantics stage.Parallel(merger.MergeCandidates, strategy.ErrorWhenAllFail, stage.FetchPopularPosts(d.PostCache), stage.FetchVideoPosts(d.PostCache), stage.FetchImagePosts(d.PostCache), )
170. 170 What’s next? Latency Budgets stage.Series(budget.Latency(100 * time.Millisecond), stage.FetchRecentlyViewedPosts(d.UserPostViews), stage.FilterRecentlyViewedPosts(), )
171. 171 What’s next? No-Code Abstraction
172. 172 What’s next? No-Code Abstraction --- name: PopularFeed stages: - name: Series stages: - name: Parallel stages: - name: FetchPopularPosts - name: FetchVideoPosts - name: FetchImagePosts - name: FetchRecentlyViewedPosts - name: FilterRecentlyViewedPosts - name: ScoreCandidares - name: SortCandidates
173. 173 What’s next? Pipelines at Runtime --- name: PopularFeed stages: - name: Series stages: - name: Parallel stages: - name: FetchPopularPosts - name: FetchVideoPosts - name: FetchImagePosts - name: FetchRecentlyViewedPosts - name: FilterRecentlyViewedPosts - name: ScoreCandidares - name: SortCandidates
174. Summary • Essential complexity describes a problem at its core, accidental complexity happens as part of solving the problem, creating unnecessary challenges; accidental complexity can be reduced • A recommendation system consists of a variety of different ranking flows with the goal to generate content that users find compelling • UNIX pipes as an inspiration to build a system of small, reusable components with the help of the single-method interface in Go • Reusability and clarity are competing concepts. This is Go: choose clarity.
175. Thank you Konrad Reiche @konradreiche u/konradreiche
176. Questions? Konrad Reiche @konradreiche u/konradreiche