Continuous Culture

An updated version of the Continuous Culture presentation, delivered at LeaseWeb's Tech Summit in Amsterdam.

Transcript

1. Continuous Culture LeaseWeb Tech Summit 4/6/2015 Scaling Engineering Culture at Etsy
2. @Ryan_Frantz #me • Senior Operations Engineer at Etsy • Built and managed teams in the past • Contributes to and helps maintain culture at Etsy
4. @Ryan_Frantz Why Culture?
6. @Ryan_Frantz Culture Constantly Develops • Communication is paramount • No divide between managers and engineers • Technology/product can attract talent, but culture keeps them
7. @Ryan_Frantz Etsy By the Numbers • 20.8 million active buyers, 1.4 million active sellers • Nearly Every Country Performing Transactions • > 700 Employees • Offices in 7 countries
8. @Ryan_Frantz Core Engineering Principles • Just Ship • Every engineer can push to prod at any time • Favor simplicity and speed to ship • Maximize learning • ‘If it moves, graph it’
9. @Ryan_Frantz Very end of 2009 Today Pushes Per Day
10. @Ryan_Frantz Continuous Culture v1 • Mike Rembetsy & Patrick McDonnell • Presented at Velocity Conference • Etsy’s engineering culture evolution 2006-2012 • Slides: http://slidesha.re/1xYxZrG • Video: http://vimeo.com/51310058
12. @Ryan_Frantz 2006 - 2008 Silos and Barriers • 4-person startup grows to employ 30 - 35 (including 15 ENG) • Siloed; little collaboration • Example: Sprouter - ‘Middleware of distrust!’ • Engineers can’t touch databases!
13. @Ryan_Frantz Management Changes • Maria Thomas from NPR promoted to CEO • Brings a clear understanding that community is very important • Prioritizes a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team
14. @Ryan_Frantz 2006-2008 Review • Downtime was an accepted fact of life • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
15. @Ryan_Frantz 2009 Internal Improvements • As teams grow, big efforts in good communication • Daily standups • DevTools team • Deployinator • lighttpd -> Apache • Network solidified
16. @Ryan_Frantz 2009 Review • Built solid foundations • Invested in human capital • Strong collaboration • Infrastructure • A lot of reflection and finding an Engineering identity
17. 2010 Renewed Energy
19. @Ryan_Frantz 2010 Standardization & Graphs • Moved to PHP & MySQL for everything • ‘If it moves, graph it’ • Ganglia, Graphite, FITB, Nagios, Naglite • Starting to use this data for work/life balance as well as technical/systems reasons
20. @Ryan_Frantz Hard Choices
22. @Ryan_Frantz 2010 Ideals • Blameless Postmortems • 1:1s as a core mgmt tool • Engineering career planning • Developer on-call • Use of A/B testing • Feature Flags & Ramp Up • Schema Change Thursday Management Ideals Engineering Ideals
23. @Ryan_Frantz 2010 Review • Reduce number of technologies used in development • Focus on technical visibility throughout the org • Developers responsible for code deploy (and on-call!) • Work/life balance focus
24. @Ryan_Frantz 2011 Tech Highlights • End of long tail legacy silo holdovers (Sprouter gone!) • Non-standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
25. @Ryan_Frantz 2011 - Organizational Changes • Senior management to become more Engineering focused • Chad to CEO • Kellan promoted to CTO • Allspaw promoted to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
26. @Ryan_Frantz 2011 Review • Year of open sourced tools! • Statsd, Logster, Deployinator, Supergrep, Schemanator • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef) • Security front and center (without negative impact!)
27. @Ryan_Frantz 2011-2012 - A Focus on Security • Security alongside Dev & Ops as being integral to culture • Applying our core principles & learnings to security • Emphasis on security being a facilitator not a blocker • Build a human and effective security organization
28. @Ryan_Frantz 2012 - Growth + Foster Our Values • Explosive growth in hiring, allow easy team transfers • Some major changes around product • Internationalization • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp
29. @Ryan_Frantz What’s a B-Corp ? • Aim to use the power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data center usage & partners • ‘Make the world more like Etsy’ - Extending the culture
30. @Ryan_Frantz 2012 - Technical Achievements • Create wholly separate Payments environment • Allows PCI compliance without disrupting the culture • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
31. @Ryan_Frantz 2013 - An Interesting Year! • Had many of the hard engineering wins taken care of • No engineer can know everything any longer • Internal tooling • Designated Ops
32. @Ryan_Frantz 2013 - Tooling • Morgue: Stores postmortem details (including IRC logs, images) • Opsweekly: Categorize and report Nagios alerts • Superbit: Allows simple querying of Vertica, Elasticsearch & big data by anyone who knows SQL • Catapult: Communicates experiments and related metrics (including conversion impact)
33. @Ryan_Frantz 2013 - Designated Ops • Visibility of ongoing projects • Spreading knowledge • Advocating for engineers
35. @Ryan_Frantz 2014 - Organizational changes • Everyone pushes on their 1st day • San Francisco office opens • Acquire & integrate A Little Market
36. @Ryan_Frantz Cultural Acquisition • Paris-based A Little Market (ALM) • Integrating another engineering culture can be tough • Language, timezone and human cultural differences • While challenging, can be very successful
37. @Ryan_Frantz 2014 - Technical Highlights • Mobile First has increasing product focus • API First: Everything is an API call, for all clients • Migrate from Splunk to ELK (Elasticsearch/Logstash/Kibana) • Technical work for quality of life: On-call sleep tracking
38. @Ryan_Frantz Mobile First • World is increasingly mobile • App stores force rethink of continuous deployment • Need to maintain Core Engineering Principles
39. @Ryan_Frantz ELK Lessons • Changing a core tool requires huge communication investment • Understanding usage patterns • Accommodating that usage and factoring into the project • Describing the purpose of the project (not just about $$) • Technology can find its way into unexpected corners of your stack!
40. @Ryan_Frantz Sleep Tracking • Ops Experiment with Jawbone UPs • Collect on-call sleep data • Analyze sleep lost when on-call • Quality of life improvements
41. @Ryan_Frantz 2015 Action Items • Culture is still king despite growth or M&A activity • It takes effort to keep it so • Ensure our API is up to the job of supporting Mobile First • Ensure core tooling changes are understood & embraced by all • Communicate our engineering culture & history to new hires
42. Credit for the awesome graph: Patrick Koch
43. @Ryan_Frantz Conclusions • Culture doesn’t come for free, it takes continuous work • Iterate & improve - Even when you think you have ‘it’ • Embrace potential disruptors like growth as opportunities to improve
44. Never. Stop. Pushing.
45. @Ryan_Frantz Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, What Went Wrong, What Went Right (Ross Snyder) Slides & video: http://bit.ly/po8zIj Etsy’s Journey to Continuous Integration for Mobile Apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean Time to Sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: 
 http://ryanfrantz.com/mtts/
46. Questions? frantz@etsy.com @Ryan_Frantz