Design a mobile app logging platform that scales for hundreds of engineers

Logging for a giant app is challenging. Besides the general challenges such as mobile network reliability, data quality or performance concern. For a giant app you have to control the log growth sustainably, ensure data interoperability across logs from teams who probably even don't know each other, avoid the tragedy of the commons, reduce cognitive load for logging, control data quality without slowing people down...

The talk shares some experience we learnt in designing a sustainable logging platform that is effective and scales. It will cover the principles, origins, evolvement, and some good practices. The talk will steer away from specific technical terms or implementation details, but the abstract design concepts should be transferrable and IMO way more important. I think it's a fun story of a platform team (well described concept from Team Topologies) in action.

Through the journey, we learn some of the values that an architect can bring (quotes from Software Architect Elevator):
- Prevent people from exercising intelligence unnecessarily.
- Act like a gardener, create context where ideas can evolve and thrive in harmony.

Transcript

1. Design a mobile app logging platform that scales for hundreds of engineers - Shuzhi Huang
2. Software engineering manager in Google. Mostly working on logging, experimentation and performance on mobile apps. Passion to empower product velocity, allow teams in big companies to move faster and more effectively. Personal Bio
3. Disclaimer: not an implementation playbook. High level lessons and principles in designing a logging platform that scales for a mobile app, taking into consideration of techniques, teams, products and culture. Preview
4. Tributes to the team and many excellent partners
5. Put yourself in the shoes of logging owner for an app
6. logger.d(TAG, “hello world”); OK, logging
9. Where are we? Default Stage 1 1
10. The app continues to grow
11. • How many times a feature is triggered? • How reliable is your connection to the server? • How long does a task take? • How many clicks to an icon? • … There are more things you need to know
12. Extended business service • Buffer into your existing http requests (e.g. CGI param) to the server. • Send a dedicated http call to existing business server. Feature Foo: { Duration:1600ms, Status: SUCCEED, Connection: WiFi … }
13. Where are we? Default Stage 2 Stage 1 1 S 1 Extension 2
15. Single responsibility: a dedicated logging service • Batches and upload when not in user critical path. • Optimize for mobile performance consideration (e.g. wifi mode, sampling).
16. Everything is just an event APP_OPEN VIEW_SHOWN SWIPE CLICK CONNECTION_START …
17. Where are we? Default Stage 2 Stage 1 1 S 1 Extension 2 4 3 Events Stage 3
18. The app continues to grow
19. Everything is just an event
20. Unmaintainable cost
21. Losing control and understanding
22. Difficulty in getting insights
23. Problems → Solutions Timer Reliability Tracker Counter Error System Health … Usages Batching Aggregation Sampling Transport
24. Problems → Problems E r r o r Timer Relia- bility Counter Dream Reality
25. A puzzled feature developer
26. Exhausted logging team(s)
27. Where are we? Default Stage 2 Stage 1 1 S 1 Extension 2 4 Fragmentation 3 Events Stage 3
28. Regain control Gatekeeping team Platform team
29. Team Topologies definition of Platform Team A grouping of other team types that provide a compelling internal product to accelerate delivery by Stream-aligned teams
30. So what does the team do? Empower the teams to do logging Maintain a healthy logging platform
31. #1 Empower the teams
32. #1 Empower the teams Logging team(s) - polish each of the layer’s solution: • UI layer, tie to the view • Business layer, declarative on tasks • System layer, automatic and performance sensitive (aggregation, sampling, buffering)
33. #1 Empower the teams Developers: think about what layer of answers you are looking for, then seek for logging tools in that layer.
34. #1 Empower the teams Product mindset and product ideology
35. #1 Empower the teams Using your services won’t make your clients less independent. Shareable SUPER Principle Understand and solve client problems, empower them. Useful Simplicity and ease of use, self-serviced, automation over toil. Easy Respect the users, cost efficient. Performant Being unreliable is worse than don’t exist. Reliable
36. #2 Maintain a healthy platform Accountability: data needs to be owned
37. #2 Maintain a healthy platform Data as a product: shared syntax, declarative, surfaced and accountable
38. #2 Maintain a healthy platform Uphold data quality: metrics, surfacing, alerting, auto-action on staleness, volume and reliability
39. #2 Maintain a healthy platform Shared binary Product A Configuration Product A dashboards, docs, tools Distributed model: different products decoupled but sharing the same code
40. How was the result? • From a stream-aligned team perspective: three-layer well-lit path, low cognitive load. No one stands in my way, no approval needed (in well-lit path). Maintain or retire my own logs. • From holistic product team or analysts team perspective: due to the shared syntax and semantics (layer), they can understand the full picture, even across teams. • From logging platform perspective: provide a 3-layer service, don’t do the operation for others, and the platform is healthy.
41. 1. Start from default 2. An extension on existing business service 3. An event-based dedicated service 4. Various organic ‘solutions’ addressing symptoms 5. Systematic platform design for the overall landscape Recap
42. 1. Start from default 2. Organic: An extension on existing business service 3. Architect: An event-based dedicated service 4. Organic: Various organic ‘solutions’ addressing symptoms 5. Architect: Systematic platform design for the overall landscape Architect Recap
43. Many other challenges
44. Thanks