Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Culture

Avatar for RyanFrantz RyanFrantz
January 14, 2015

Continuous Culture

Culture is always evolving and doesn't come for free. This is the story of Etsy's journey from 4-person startup to an international organization and the culture that we all work to foster.

Avatar for RyanFrantz

RyanFrantz

January 14, 2015
Tweet

More Decks by RyanFrantz

Other Decks in Business

Transcript

  1. @Ryan_Frantz #me • Ryan Frantz • Senior Operations Engineer at

    Etsy (Remote) • Built and managed teams in the past • Contributes to and helps maintain culture at Etsy
  2. @Ryan_Frantz By the Numbers (FY 2013) • Gross Marketplace Sales

    (GMS) $1.35 Billion • 44 million members, 1.2 million active sellers • 26 million active listings • Nearly Every Country Performing Transactions • > 600 Employees
  3. @Ryan_Frantz There WILL be a test… • Culture constantly develops

    • Communication is paramount • No divide between managers and engineers • Technology/product can attract people, but culture keeps them
  4. @Ryan_Frantz Core Engineering Principles • Just Ship • Every engineer

    can push to prod at any time • Favor simplicity and speed to ship • Maximize learning • ‘If it moves, graph it’
  5. @Ryan_Frantz Continuous Culture v1 • Mike Rembetsy & Patrick McDonnell

    • Presented at Velocity Conference • Etsy’s engineering culture evolution 2006-2012 • Slides: http://slidesha.re/1xYxZrG • Video: http://vimeo.com/51310058
  6. @Ryan_Frantz 2006 - 2008 Silos and Barriers • 4-person startup

    grows to employ 30 - 35 (including 15 ENG) • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases
  7. @Ryan_Frantz Management Changes • Maria Thomas from NPR promoted to

    CEO • Brings a clear understanding that community is very important • Prioritizes a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • “This siloed culture cannot work, we need to start over”
  8. @Ryan_Frantz 2006-2008 Takeaways • Downtime was an accepted fact of

    life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
  9. @Ryan_Frantz 2009 Internal Improvements • As teams grow, big efforts

    in good communication • Daily standups • DevTools team • Deployinator • lighttpd -> Apache • Network solidified
  10. @Ryan_Frantz 2009 Takeaways • Built solid foundations • Invested in

    human capital • Strong collaboration • Infrastructure • A lot of reflection and finding an Engineering identity
  11. @Ryan_Frantz 2010 Standardization & Graphs • Moved to PHP &

    MySQL for everything • ‘If it moves, graph it’ • Ganglia, Graphite, FITB, Nagios, Naglite • Starting to use this data for work/life balance as well as technical/systems reasons
  12. @Ryan_Frantz 2010 Ideals • Blameless Postmortems • 1:1s as a

    core mgmt tool • Engineering career planning • Developer on-call • Use of A/B testing • Feature Flags & Ramp Up • Schema Change Thursday Management Ideals Engineering Ideals
  13. @Ryan_Frantz 2010 Takeaways • Reduce number of technologies used in

    development • Focus on technical visibility throughout the org • Developers responsible for code deploy (and on-call!) • Work/life balance focus
  14. @Ryan_Frantz 2011 Tech Highlights • End of long tail legacy

    silo holdovers (Sprouter gone!) • Non-standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
  15. @Ryan_Frantz 2011 - Organizational Changes • Senior management to become

    more Engineering focused • Chad to CEO • Kellan promoted to CTO • Allspaw promoted to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
  16. @Ryan_Frantz 2011 Takeaways • Year of open sourced tools! •

    Statsd, Logster, Deployinator, Supergrep, Schemanator • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef) • Security front and center • Security without negative impact to culture?
  17. @Ryan_Frantz 2011-2012 - A Focus on Security • Security alongside

    Dev & Ops as being integral to culture • Applying our core principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often enforced with terrible cultural impact • Build a human and effective security organization
  18. @Ryan_Frantz 2012 - Growth + Foster Our Values • Explosive

    growth in hiring, allow easy team transfers • Some major changes around product • Internationalization • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp
  19. @Ryan_Frantz What’s a B-Corp ? • Aim to use the

    power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data center usage & partners • ‘Make the world more like Etsy’ - Extending the culture
  20. @Ryan_Frantz 2012 - Technical Achievements • Create wholly separate Payments

    environment • Allows PCI compliance without disrupting the culture • Interface with the web stack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
  21. @Ryan_Frantz 2012 Takeaways • Do what’s needed to sustain long

    term goals & not just keep the lights on • More headcount allows us to take chances • Focus on social/environmental impact, internally & externally with communities • Open source all of the things
  22. @Ryan_Frantz 2013 - An Interesting Year! • Had many of

    the hard engineering wins taken care of • Time to focus internally • No engineer can know everything any longer • Internal tooling • Designated Ops
  23. @Ryan_Frantz 2013 - Tooling • Morgue: Stores postmortem details (including

    IRC logs, images) • Opsweekly: Categorize and report Nagios alerts • Superbit: Allows simple querying of Vertica, Elasticsearch & big data by anyone who knows SQL • Catapult: Communicates experiments and related metrics (including conversion impact) • Begin a refocus on a Mobile/API First product vision
  24. @Ryan_Frantz 2013 - Designated Ops • Visibility of ongoing projects

    • Spreading knowledge • Advocating for engineers • Assignments rotate annually
  25. @Ryan_Frantz 2013 Takeaways • Democratization of data is made easier

    with tooling that levels access and allows interrogation by ALL • Designated Ops • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future
  26. @Ryan_Frantz 2014 - Organizational changes • Everyone pushes on their

    1st day • Yearly planning is restructured • San Francisco office opens • Acquire & integrate A Little Market
  27. @Ryan_Frantz Cultural Acquisition • Paris-based A Little Market (ALM) •

    Integrating another engineering culture can be tough • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate
  28. @Ryan_Frantz 2014 - Technical • Migrate from Splunk to ELK

    (Elasticsearch/Logstash/Kibana) • Mobile First has increasing product focus • Mobile CI infrastructure embedded & ramped up • API First is huge effort and development push • Technical work for quality of life: On-call sleep tracking
  29. @Ryan_Frantz ELK Lessons • Changing a core tool requires huge

    communication investment • Understanding usage patterns • Accommodating that usage and factoring into the project • Describing the purpose of the project (not just about $$) • Technology can find its way into unexpected corners of your stack!
  30. @Ryan_Frantz Mobile First • Applying your principles and culture to

    the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Use API to enable feature flag-driven native apps
  31. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  32. @Ryan_Frantz API First • Supporting the Mobile First push &

    diversity of clients • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging
  33. @Ryan_Frantz Why This Approach? • Continuous integration, Continuous Delivery •

    Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases!
  34. Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  35. @Ryan_Frantz Sleep Tracking • Experiment with Jawbone UPs & Ops

    • Collect on-call sleep data • Analyze sleep lost when on-call • Focus on data for quality of life
  36. @Ryan_Frantz 2014 Takeaways • Another year of big growth +

    M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & communication • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all
  37. @Ryan_Frantz 2015 Action Items • Culture is still king despite

    growth or M&A activity • It takes effort to keep it so however • Ensure our API is up to the job of supporting Mobile First • Ensure core tooling changes are understood & embraced by all • Communicate our engineering culture & history to new hires
  38. @Ryan_Frantz Conclusions • Culture doesn’t come for free, it takes

    continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture
  39. @Ryan_Frantz Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick

    McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, What Went Wrong, What Went Right (Ross Snyder) Slides & video: http://bit.ly/po8zIj Etsy’s Journey to Continuous Integration for Mobile Apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean Time to Sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: 
 http://ryanfrantz.com/mtts/