Contiuously Deploying Culture 2.0 - Agile Ísland

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=47 Rich Smith
November 09, 2014

Contiuously Deploying Culture 2.0 - Agile Ísland

Presentation discussing the story of engineering culture at Etsy and the lessons learnt of maintaining a genuine and engaging culture in a rapidly growing technology company.

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=128

Rich Smith

November 09, 2014
Tweet

Transcript

  1. Continuously Deploying Culture 2.0 Rich Smith Director of Security, Etsy

    @iodboi Agile Iceland 2014 A Story Of Scaling Engineering Culture at Etsy
  2. @iodboi Who Am I ? • Rich Smith, Brooklyn NYC

    • Director of Security Engineering at Etsy • Cover the AppSec, NetSec, Risk Engineering teams • Also building the security organisation & security culture • Co-Founder of here in Iceland !
  3. @iodboi WTF is ! • Online community for handcrafted &

    vintage items • Human scale manufacturing • Marketplace for small businesses
  4. @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35

    Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
  5. @iodboi Core Engineering Principles • Empower the edges • Trustworthy

    not trusted • Every engineer can push to prod at any time • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you
  6. @iodboi Very end of 2009 Today Pushes Per Day

  7. But How Did We Get There?

  8. @iodboi What’s Etsy’s Story of Engineering Culture?

  9. @iodboi Continuously Deploying Culture v1 • Mike Rembetsy & Patrick

    McDonnell • Gave a talk at the Velocity conferences in 2011/12 • Etsy’s engineering culture evolution 2006-2011/12 • Slides here: http://slidesha.re/1xYxZrG 
 Watch it here: http://vimeo.com/51310058 • Today we are extending those lessons up to the
  10. Disclaimer A + B != Culture

  11. None
  12. None
  13. @iodboi 2006 - 2008 Silos and Barriers • Etsy 4

    person startup grows to employ 30 - 35 FTE’s • Around 15 engineers • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases
  14. @iodboi Management Changes • Maria Thomas from NPR promoted to

    CEO • Brings a clear understanding that community is very important • Prioritises a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • ‘This Silo’d culture cannot work, we need to start over’
  15. @iodboi 2006-2008 Takeaways • Downtime was an accepted fact of

    life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
  16. @iodboi 2008 End of Year Snapshot GMS Employees Engineers $87.5

    Million 35 15
  17. @iodboi 2009 Internal Improvements • As teams grow, big efforts

    in good communication • Daily standups begun • Much better cross-team collaboration between Ops & others • Network solidified and provided basis for future growth • Moved from Downtown Brooklyn to DUMBO
  18. @iodboi 2009 Takeaways • Big growth year • Built solid

    foundations: • Infrastructure • Invested in human capital • ‘DevOps’ culture begins in earnest ….. • A lot of reflection and finding an Engineering identity
  19. @iodboi 2009 End of Year Snapshot GMS $180.6 Million

  20. 2010 Renewed Energy

  21. None
  22. @iodboi 2010 Standardisation & Graphs • Moved to PHP &

    MySQL for * • It almost doesn’t matter what you choose, just stick to it • ‘If it moves Graph it’ • Graphite, Ganglia, FITB, Weathermap, Nagios, Naglite ….. • Starting to use this data for work/life balance as well as technical/systems reasons
  23. **

  24. None
  25. @iodboi 2010 Following our ideals ….. • Blameless PostMortems •

    1:1’s as a core mgmt tool • Eng career planning (Reverb) • Accept failures, but not low standards • Developer on-call • Use of A/B testing • Lots of Prototypes • FeatureFlags & Ramp Up Management Ideals Engineering Ideals
  26. @iodboi 2010 Takeaways • Reduce number of technologies used in

    development • Focus on technical visibility throughout the org • Developers responsible for code release (start of DevOps) • Member support rotations for all • Work hard at work/life balance & have data to support
  27. @iodboi 2010 End of Year Snapshot GMS $314.3 Million

  28. @iodboi 2011 Tech highlights • End of long tail legacy

    silo holdovers (Sprouter gone!) • Non-Standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
  29. @iodboi 2011 - Organisational Changes • Snr. management to become

    more Engineering focused • Chad to CEO • Kellan to CTO • Allspaw to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
  30. @iodboi 2011 Takeaways • Year of the Open Sourced tool

    • Statsd, Logster, Deployinator, Supergrep, Schemanator …. • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef everything) • Security starts becoming a 1st class citizen • ‘Security Culture at Etsy’ begins to be chased & discussed
  31. @iodboi 2011 End of Year Snapshot GMS $525.6 Million

  32. Let’s catch our breath!

  33. @iodboi 2011-2012 - A Focus on Security • Security alongside

    Dev & Ops as being integral to culture • Applying our ‘DevOps’ principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often ‘enforced’ with terrible cultural impact • Build a human and effective security organization
  34. @iodboi 2012 - Growth + Foster Our Values • Explosive

    growth in hiring, allow easy transfers • Some major changes around product • Increased focus on community • Internationalisation • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp - not just the bottom line
  35. @iodboi What’s a B-Corp ? • Aim to use the

    power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data centre usage & partners • ‘Make the world more like Etsy’ - Extending the culture
  36. @iodboi 2012 - Technical Achievements • Create wholly separate payments

    environment • Allows PCI compliance without disrupting the culture • Interface with the webstack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
  37. @iodboi 2012 Takeaways • Do what’s needed to sustain long

    term & not just keep the lights on • More headcount than required allows us to take chances • Focus on info exchange, internally & externally with communities • Open source all of the things
  38. @iodboi 2012 End of Year Snapshot GMS $895.1 Million

  39. @iodboi 2012 Action Items • Security is integral DevOps lifecycle

    and culture • Know when to flick sights from short to longer term goals • Pursue dynamic engineering resource allocation • Do not allow increasing org size to dictate culture
  40. @iodboi 2013 - An Interesting Year! • Had many of

    the hard engineering wins taken care of … • Time to focus internally • No engineer can know everything any longer • Need to maintain the culture of transparency & trust • Really was the year of internal tooling to achieve this
  41. @iodboi 2013 - Technical • Morgue tool created to capture

    and aid postmortems • Moved to Vertica for BI data & metrics • Superbit allowed simple querying of Vertica & big data by anyone who knows SQL • Catapult launched to relate metrics to experiments • Begin a refocus on a Mobile/API First product vision
  42. @iodboi 2013 Takeaways • Democratisation of data is made easier

    with tooling that levels access and allows interrogation by ALL • Conscious effort on internal tooling to minimize the pain of large & complex stack • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future
  43. @iodboi 2013 End of Year Snapshot GMS Employees Engineers $1.35

    Billion 615 >33%
  44. @iodboi 2013 Action Items • Datasets grow, evaluate how they

    can be accessed, evaluated and contextualized • Have you reached a point where no one can know everything? • While tooling can’t create culture it can help you support it • Be free to apply your culture in new ways • Inward focus cannot lead to outward blindness, tech changes fast
  45. None
  46. @iodboi 2014 - Organisational changes • Everyone pushes on their

    1st day, not just Engineers • Yearly planning is restructured • Take account of a growing Etsy • San Francisco opens as 1st non-Brooklyn Eng hub • Acquire & integrate A Little Market with Etsy
  47. @iodboi Cultural Acquisition • As part of growth, Paris based

    A Little Market acquired • Integrating another engineering culture can be tough • Etsy’s culture is ‘different’ & this can be a big step • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate
  48. @iodboi 2014 - Technical • Move away from Splunk and

    to ElasticSearch/Logstash/ Kibana (ELK) • Mobile CI infrastructure embedded & ramped up • API First a huge effort and development push • Mobile First as an increasing product focus • Technical work for quality of life - on-call sleep tracking
  49. @iodboi Logstash Lessons • Replacing Splunk with LogStash taught many

    lessons • Changing of a core tool require huge comms investment • Without it enclaves & silos can form to resist change • Explain the whys not just in terms of technicals or $$ • Fully understanding all use cases, not just the main ones • Don’t settle for a half complete end goal, go the distance
  50. @iodboi API First • Supporting the Mobile First push &

    diversity of clients • No longer assume LAMP, decoupling required • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging
  51. @iodboi Mobile First • Applying your principles and culture to

    the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Still use API to enable feature flag driven native apps
  52. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  53. @iodboi Why This Approach? • Continuous integration, Continuous Delivery •

    Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases!
  54. Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  55. @iodboi Sleep Tracking • Experiment with fitbands & Ops •

    Collect sleep data for on-call • Analyse in a variety of manners • Sleep lost when on-call/pagerduty • Alert on VPN/SSH logins while asleep • Focus on data for quality of life
  56. @iodboi 2014 Takeaways • Another year of big growth, also

    now M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & comms • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all
  57. @iodboi 2014 Action Items • Culture is still king despite

    growth or M&A activity • It takes effort to keep it so however • Ensure your API is up to the job of supporting Mobile 1st • Ensure core tooling changes are understood & embraced by all • Communicate your Eng culture & history to new hires
  58. Credit for the awesome graph: Patrick Koch

  59. @iodboi Conclusions • Culture doesn’t come for free, it takes

    continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture • Get smart and use them to test, support and improve it
  60. Never. Stop. Pushing.

  61. @iodboi Questions?

  62. @iodboi Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick

    McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, what went wrong, what went right (Ross Snyder) Slides & Video: http://bit.ly/po8zIj Etsy’s journey to continuous integration for mobile apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean time to sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: 
 http://ryanfrantz.com/mtts/