Share
Embed
Contiuously Deploying Culture 2.0 - Agile Ísland
Presentation discussing the story of engineering culture at Etsy and the lessons learnt of maintaining a genuine and engaging culture in a rapidly growing technology company.
Rich Smith
November 09, 2014
Transcript
62 slides
-
Continuously Deploying Culture 2.0 Rich Smith Director of Security, Etsy
@iodboi Agile Iceland 2014 A Story Of Scaling Engineering Culture at Etsy -
@iodboi Who Am I ? • Rich Smith, Brooklyn NYC
• Director of Security Engineering at Etsy • Cover the AppSec, NetSec, Risk Engineering teams • Also building the security organisation & security culture • Co-Founder of here in Iceland ! -
@iodboi WTF is ! • Online community for handcrafted &
vintage items • Human scale manufacturing • Marketplace for small businesses -
@iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35
Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries -
@iodboi Core Engineering Principles • Empower the edges • Trustworthy
not trusted • Every engineer can push to prod at any time • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you -
@iodboi Continuously Deploying Culture v1 • Mike Rembetsy & Patrick
McDonnell • Gave a talk at the Velocity conferences in 2011/12 • Etsy’s engineering culture evolution 2006-2011/12 • Slides here: http://slidesha.re/1xYxZrG Watch it here: http://vimeo.com/51310058 • Today we are extending those lessons up to the -
@iodboi 2006 - 2008 Silos and Barriers • Etsy 4
person startup grows to employ 30 - 35 FTE’s • Around 15 engineers • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases -
@iodboi Management Changes • Maria Thomas from NPR promoted to
CEO • Brings a clear understanding that community is very important • Prioritises a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • ‘This Silo’d culture cannot work, we need to start over’ -
@iodboi 2006-2008 Takeaways • Downtime was an accepted fact of
life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just! -
@iodboi 2009 Internal Improvements • As teams grow, big efforts
in good communication • Daily standups begun • Much better cross-team collaboration between Ops & others • Network solidified and provided basis for future growth • Moved from Downtown Brooklyn to DUMBO -
@iodboi 2009 Takeaways • Big growth year • Built solid
foundations: • Infrastructure • Invested in human capital • ‘DevOps’ culture begins in earnest ….. • A lot of reflection and finding an Engineering identity -
@iodboi 2010 Standardisation & Graphs • Moved to PHP &
MySQL for * • It almost doesn’t matter what you choose, just stick to it • ‘If it moves Graph it’ • Graphite, Ganglia, FITB, Weathermap, Nagios, Naglite ….. • Starting to use this data for work/life balance as well as technical/systems reasons -
@iodboi 2010 Following our ideals ….. • Blameless PostMortems •
1:1’s as a core mgmt tool • Eng career planning (Reverb) • Accept failures, but not low standards • Developer on-call • Use of A/B testing • Lots of Prototypes • FeatureFlags & Ramp Up Management Ideals Engineering Ideals -
@iodboi 2010 Takeaways • Reduce number of technologies used in
development • Focus on technical visibility throughout the org • Developers responsible for code release (start of DevOps) • Member support rotations for all • Work hard at work/life balance & have data to support -
@iodboi 2011 Tech highlights • End of long tail legacy
silo holdovers (Sprouter gone!) • Non-Standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software -
@iodboi 2011 - Organisational Changes • Snr. management to become
more Engineering focused • Chad to CEO • Kellan to CTO • Allspaw to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability -
@iodboi 2011 Takeaways • Year of the Open Sourced tool
• Statsd, Logster, Deployinator, Supergrep, Schemanator …. • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef everything) • Security starts becoming a 1st class citizen • ‘Security Culture at Etsy’ begins to be chased & discussed -
@iodboi 2011-2012 - A Focus on Security • Security alongside
Dev & Ops as being integral to culture • Applying our ‘DevOps’ principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often ‘enforced’ with terrible cultural impact • Build a human and effective security organization -
@iodboi 2012 - Growth + Foster Our Values • Explosive
growth in hiring, allow easy transfers • Some major changes around product • Increased focus on community • Internationalisation • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp - not just the bottom line -
@iodboi What’s a B-Corp ? • Aim to use the
power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data centre usage & partners • ‘Make the world more like Etsy’ - Extending the culture -
@iodboi 2012 - Technical Achievements • Create wholly separate payments
environment • Allows PCI compliance without disrupting the culture • Interface with the webstack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers -
@iodboi 2012 Takeaways • Do what’s needed to sustain long
term & not just keep the lights on • More headcount than required allows us to take chances • Focus on info exchange, internally & externally with communities • Open source all of the things -
@iodboi 2012 Action Items • Security is integral DevOps lifecycle
and culture • Know when to flick sights from short to longer term goals • Pursue dynamic engineering resource allocation • Do not allow increasing org size to dictate culture -
@iodboi 2013 - An Interesting Year! • Had many of
the hard engineering wins taken care of … • Time to focus internally • No engineer can know everything any longer • Need to maintain the culture of transparency & trust • Really was the year of internal tooling to achieve this -
@iodboi 2013 - Technical • Morgue tool created to capture
and aid postmortems • Moved to Vertica for BI data & metrics • Superbit allowed simple querying of Vertica & big data by anyone who knows SQL • Catapult launched to relate metrics to experiments • Begin a refocus on a Mobile/API First product vision -
@iodboi 2013 Takeaways • Democratisation of data is made easier
with tooling that levels access and allows interrogation by ALL • Conscious effort on internal tooling to minimize the pain of large & complex stack • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future -
@iodboi 2013 End of Year Snapshot GMS Employees Engineers $1.35
Billion 615 >33% -
@iodboi 2013 Action Items • Datasets grow, evaluate how they
can be accessed, evaluated and contextualized • Have you reached a point where no one can know everything? • While tooling can’t create culture it can help you support it • Be free to apply your culture in new ways • Inward focus cannot lead to outward blindness, tech changes fast -
@iodboi 2014 - Organisational changes • Everyone pushes on their
1st day, not just Engineers • Yearly planning is restructured • Take account of a growing Etsy • San Francisco opens as 1st non-Brooklyn Eng hub • Acquire & integrate A Little Market with Etsy -
@iodboi Cultural Acquisition • As part of growth, Paris based
A Little Market acquired • Integrating another engineering culture can be tough • Etsy’s culture is ‘different’ & this can be a big step • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate -
@iodboi 2014 - Technical • Move away from Splunk and
to ElasticSearch/Logstash/ Kibana (ELK) • Mobile CI infrastructure embedded & ramped up • API First a huge effort and development push • Mobile First as an increasing product focus • Technical work for quality of life - on-call sleep tracking -
@iodboi Logstash Lessons • Replacing Splunk with LogStash taught many
lessons • Changing of a core tool require huge comms investment • Without it enclaves & silos can form to resist change • Explain the whys not just in terms of technicals or $$ • Fully understanding all use cases, not just the main ones • Don’t settle for a half complete end goal, go the distance -
@iodboi API First • Supporting the Mobile First push &
diversity of clients • No longer assume LAMP, decoupling required • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging -
@iodboi Mobile First • Applying your principles and culture to
the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Still use API to enable feature flag driven native apps -
Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓
✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ? -
@iodboi Why This Approach? • Continuous integration, Continuous Delivery •
Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases! -
Single release Many releases 50K LOC/month Few opportunities for failure
Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve written per release Imagine that we’ll write -
@iodboi Sleep Tracking • Experiment with fitbands & Ops •
Collect sleep data for on-call • Analyse in a variety of manners • Sleep lost when on-call/pagerduty • Alert on VPN/SSH logins while asleep • Focus on data for quality of life -
@iodboi 2014 Takeaways • Another year of big growth, also
now M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & comms • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all -
@iodboi 2014 Action Items • Culture is still king despite
growth or M&A activity • It takes effort to keep it so however • Ensure your API is up to the job of supporting Mobile 1st • Ensure core tooling changes are understood & embraced by all • Communicate your Eng culture & history to new hires -
@iodboi Conclusions • Culture doesn’t come for free, it takes
continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture • Get smart and use them to test, support and improve it -
@iodboi Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick
McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, what went wrong, what went right (Ross Snyder) Slides & Video: http://bit.ly/po8zIj Etsy’s journey to continuous integration for mobile apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean time to sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: http://ryanfrantz.com/mtts/