Continuous Culture

Culture is always evolving and doesn't come for free. This is the story of Etsy's journey from 4-person startup to an international organization and the culture that we all work to foster.

Transcript

1. Continuous Culture USPTO DevOps Meetup 1/14/2015 Scaling Engineering Culture at Etsy
2. @Ryan_Frantz #me • Ryan Frantz • Senior Operations Engineer at Etsy (Remote) • Built and managed teams in the past • Contributes to and helps maintain culture at Etsy
4. @Ryan_Frantz By the Numbers (FY 2013) • Gross Marketplace Sales (GMS) $1.35 Billion • 44 million members, 1.2 million active sellers • 26 million active listings • Nearly Every Country Performing Transactions • > 600 Employees
5. @Ryan_Frantz There WILL be a test… • Culture constantly develops • Communication is paramount • No divide between managers and engineers • Technology/product can attract people, but culture keeps them
6. @Ryan_Frantz Core Engineering Principles • Just Ship • Every engineer can push to prod at any time • Favor simplicity and speed to ship • Maximize learning • ‘If it moves, graph it’
7. @Ryan_Frantz Very end of 2009 Today Pushes Per Day
8. @Ryan_Frantz Continuous Culture v1 • Mike Rembetsy & Patrick McDonnell • Presented at Velocity Conference • Etsy’s engineering culture evolution 2006-2012 • Slides: http://slidesha.re/1xYxZrG • Video: http://vimeo.com/51310058
9. Disclaimer A + B != Culture
10. But How Did We Get There?
12. @Ryan_Frantz 2006 - 2008 Silos and Barriers • 4-person startup grows to employ 30 - 35 (including 15 ENG) • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases
13. @Ryan_Frantz Management Changes • Maria Thomas from NPR promoted to CEO • Brings a clear understanding that community is very important • Prioritizes a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • “This siloed culture cannot work, we need to start over”
14. @Ryan_Frantz 2006-2008 Takeaways • Downtime was an accepted fact of life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
15. @Ryan_Frantz 2008 End of Year Snapshot GMS Employees Engineers $87.5 Million 35 15
16. @Ryan_Frantz 2009 Internal Improvements • As teams grow, big efforts in good communication • Daily standups • DevTools team • Deployinator • lighttpd -> Apache • Network solidified
17. @Ryan_Frantz 2009 Takeaways • Built solid foundations • Invested in human capital • Strong collaboration • Infrastructure • A lot of reflection and finding an Engineering identity
18. @Ryan_Frantz 2009 End of Year Snapshot GMS $180.6 Million
19. 2010 Renewed Energy
21. @Ryan_Frantz 2010 Standardization & Graphs • Moved to PHP & MySQL for everything • ‘If it moves, graph it’ • Ganglia, Graphite, FITB, Nagios, Naglite • Starting to use this data for work/life balance as well as technical/systems reasons
23. @Ryan_Frantz 2010 Ideals • Blameless Postmortems • 1:1s as a core mgmt tool • Engineering career planning • Developer on-call • Use of A/B testing • Feature Flags & Ramp Up • Schema Change Thursday Management Ideals Engineering Ideals
24. @Ryan_Frantz 2010 Takeaways • Reduce number of technologies used in development • Focus on technical visibility throughout the org • Developers responsible for code deploy (and on-call!) • Work/life balance focus
25. @Ryan_Frantz 2010 End of Year Snapshot GMS $314.3 Million
26. @Ryan_Frantz 2011 Tech Highlights • End of long tail legacy silo holdovers (Sprouter gone!) • Non-standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
27. @Ryan_Frantz 2011 - Organizational Changes • Senior management to become more Engineering focused • Chad to CEO • Kellan promoted to CTO • Allspaw promoted to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
28. @Ryan_Frantz 2011 Takeaways • Year of open sourced tools! • Statsd, Logster, Deployinator, Supergrep, Schemanator • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef) • Security front and center • Security without negative impact to culture?
29. @Ryan_Frantz 2011 End of Year Snapshot GMS $525.6 Million
30. 7th Inning Stretch
31. @Ryan_Frantz 2011-2012 - A Focus on Security • Security alongside Dev & Ops as being integral to culture • Applying our core principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often enforced with terrible cultural impact • Build a human and effective security organization
32. @Ryan_Frantz 2012 - Growth + Foster Our Values • Explosive growth in hiring, allow easy team transfers • Some major changes around product • Internationalization • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp
33. @Ryan_Frantz What’s a B-Corp ? • Aim to use the power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data center usage & partners • ‘Make the world more like Etsy’ - Extending the culture
34. @Ryan_Frantz 2012 - Technical Achievements • Create wholly separate Payments environment • Allows PCI compliance without disrupting the culture • Interface with the web stack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
35. @Ryan_Frantz 2012 Takeaways • Do what’s needed to sustain long term goals & not just keep the lights on • More headcount allows us to take chances • Focus on social/environmental impact, internally & externally with communities • Open source all of the things
36. @Ryan_Frantz 2012 End of Year Snapshot GMS $895.1 Million
37. @Ryan_Frantz 2013 - An Interesting Year! • Had many of the hard engineering wins taken care of • Time to focus internally • No engineer can know everything any longer • Internal tooling • Designated Ops
38. @Ryan_Frantz 2013 - Tooling • Morgue: Stores postmortem details (including IRC logs, images) • Opsweekly: Categorize and report Nagios alerts • Superbit: Allows simple querying of Vertica, Elasticsearch & big data by anyone who knows SQL • Catapult: Communicates experiments and related metrics (including conversion impact) • Begin a refocus on a Mobile/API First product vision
39. @Ryan_Frantz 2013 - Designated Ops • Visibility of ongoing projects • Spreading knowledge • Advocating for engineers • Assignments rotate annually
40. @Ryan_Frantz 2013 Takeaways • Democratization of data is made easier with tooling that levels access and allows interrogation by ALL • Designated Ops • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future
41. @Ryan_Frantz 2013 End of Year Snapshot GMS Employees Engineers $1.35 Billion 615 >33%
43. @Ryan_Frantz 2014 - Organizational changes • Everyone pushes on their 1st day • Yearly planning is restructured • San Francisco office opens • Acquire & integrate A Little Market
44. @Ryan_Frantz Cultural Acquisition • Paris-based A Little Market (ALM) • Integrating another engineering culture can be tough • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate
45. @Ryan_Frantz 2014 - Technical • Migrate from Splunk to ELK (Elasticsearch/Logstash/Kibana) • Mobile First has increasing product focus • Mobile CI infrastructure embedded & ramped up • API First is huge effort and development push • Technical work for quality of life: On-call sleep tracking
46. @Ryan_Frantz ELK Lessons • Changing a core tool requires huge communication investment • Understanding usage patterns • Accommodating that usage and factoring into the project • Describing the purpose of the project (not just about $$) • Technology can find its way into unexpected corners of your stack!
47. @Ryan_Frantz Mobile First • Applying your principles and culture to the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Use API to enable feature flag-driven native apps
48. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓ ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
49. @Ryan_Frantz API First • Supporting the Mobile First push & diversity of clients • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging
50. @Ryan_Frantz Why This Approach? • Continuous integration, Continuous Delivery • Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases!
51. Single release Many releases 50K LOC/month Few opportunities for failure
 Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
52. @Ryan_Frantz Sleep Tracking • Experiment with Jawbone UPs & Ops • Collect on-call sleep data • Analyze sleep lost when on-call • Focus on data for quality of life
53. @Ryan_Frantz 2014 Takeaways • Another year of big growth + M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & communication • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all
54. @Ryan_Frantz 2015 Action Items • Culture is still king despite growth or M&A activity • It takes effort to keep it so however • Ensure our API is up to the job of supporting Mobile First • Ensure core tooling changes are understood & embraced by all • Communicate our engineering culture & history to new hires
55. Credit for the awesome graph: Patrick Koch
56. @Ryan_Frantz Conclusions • Culture doesn’t come for free, it takes continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture
57. Never. Stop. Pushing.
58. @Ryan_Frantz Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, What Went Wrong, What Went Right (Ross Snyder) Slides & video: http://bit.ly/po8zIj Etsy’s Journey to Continuous Integration for Mobile Apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean Time to Sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: 
 http://ryanfrantz.com/mtts/
59. Questions? frantz@etsy.com @Ryan_Frantz