Enabling Engineering Productivity at the Financial Times

The Financial Times has embraced autonomy in our development teams, and as a result we move pretty fast - over 30,000 releases in a year from a development team of around 250. Because we have a wide variety of technical architectures, processes and team structures, we need to make sure that all those autonomous teams build in security, fault detection, and cost efficiency.

This is the responsibility of the Engineering Enablement group at the FT: teams who provide tooling and services for common capabilities like DNS, content delivery, cloud provisioning, observability, etc.; manage relationships with vendors; and provide insights and oversight to all our development teams.

To do this effectively, we have adopted principles and practices that centre on making sure teams don’t need to wait for us to do things, and we have stayed constantly focused on what our customer teams need from us.

In these slides, I discuss our approach and share what you can learn from our experiences working in this area for the last three years.

Transcript

1. Enabling Engineering Productivity at the Financial Times Sarah Wells Technical Director for Engineering Enablement @sarahjwells
2. Enabling Engineering Productivity at the Financial Times Sarah Wells Technical Director for Engineering Enablement @sarahjwells
3. Changes to production at the FT
4. It’s only an experiment if it can fail “Experiments: the Good, the Bad and the Beautiful” by Linda Rising
5. • Delivery lead time • Deployment frequency • Change fail rate • Time to restore service
6. “Full stack” stops somewhere!
7. Enter Engineering Enablement Our aim is to standardise, simplify and advise, supporting the FT's product development teams so that they can deliver value quickly, securely and scalably.
8. In Team Topologies terms, we are platform and enabling teams
10. Pave the road Guardrails not fences
11. A cautionary tale…
12. FT Platform impact on server provisioning time
13. You needed sudo access to deploy code for the first time to a newly provisioned server
14. Autonomous teams can choose something else http:/ /matt.chadburn.co.uk/notes/teams-as- services.html
15. Teams chose other options: • Heroku • Containers
16. Internal teams are service providers now
17. Guardrails not fences
19. People shouldn’t need to read the guardrails
20. You need to do this to spin up AWS resources
24. Evolving the guardrails
25. Our Tech Governance Group
26. Pave the road
28. The golden path: An opinionated and supported way of doing things
29. “Supported”
30. Principles for building the golden path
31. Valuable Has obvious value to engineers Should we provide this capability?
32. Make sure someone is signed up to use the thing you are building
33. Owned and supported It won’t disappear under people Transparent usage and cost insights It’s clear who is using it and how much their bill would be Can people rely on it?
35. Discoverable Engineers can find out it exists Documented Step by step guides, explainers, reference docs all exist Self service You can solve your problem yourself Can people use this without costly co-ordination?
36. The Tech Hub
37. “Lots of manual PR approvals, which delays time to release. E.g. approvals for DNS repo”
38. Only added lines Only removed lines Modified but line count the same 220 (47%) 71 (15%) 141 (30%) 117 (25%) approved without any comments
39. • Automatically approve simple stuff • Send DMARC/DKIM/MX changes to Cyber Security team for approval • Check for common mistakes • Look for modifications that are benign and auto- approve • Use biz-ops to find a suitable team member that can peer review
40. Easy to use Documentation guides new users - an ‘on- ramp’ Consistent developer experience If you’ve used other capabilities, this should be recognisably similar Will people get stuck?
42. Independent yet composable You can use it on its own or combine it with other capabilities Automation friendly APIs, SDKs, CLIs Can people use it in ways we didn’t expect?
44. Safe to use Sensible defaults, small blast radius Secure and compliant Security issues are fixed for you, capabilities comply with our policies Reliable Suitable levels of availability, scalability and performance Does it guide people to do the right thing?
45. The Biz Ops graph
46. Runbooks are extracted to S3
47. Valuable Owned and supported Transparent usage and cost insights Discoverable Documented Self service Easy to use Consistent developer experience Independent yet composable Automation friendly Safe to use Secure and compliant Reliable
48. Charity Majors: https:/ /charity.wtf/ 2018/12/02/software-sprawl-the-golden- path-and-scaling-teams-with-agency/ Galo Navarro: https:/ /srvaroa.github.io/paas/ infrastructure/platform/kubernetes/cloud/ 2020/01/02/talk-how-to-build-a-paas- for-1500-engineers.html https:/ /backstage.io/
49. Thank you @sarahjwells