Presentation discussing the story of engineering culture at Etsy and the lessons learnt of maintaining a genuine and engaging culture in a rapidly growing technology company.
• Director of Security Engineering at Etsy • Cover the AppSec, NetSec, Risk Engineering teams • Also building the security organisation & security culture • Co-Founder of here in Iceland !
Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
McDonnell • Gave a talk at the Velocity conferences in 2011/12 • Etsy’s engineering culture evolution 2006-2011/12 • Slides here: http://slidesha.re/1xYxZrG Watch it here: http://vimeo.com/51310058 • Today we are extending those lessons up to the
person startup grows to employ 30 - 35 FTE’s • Around 15 engineers • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases
CEO • Brings a clear understanding that community is very important • Prioritises a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • ‘This Silo’d culture cannot work, we need to start over’
life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
in good communication • Daily standups begun • Much better cross-team collaboration between Ops & others • Network solidified and provided basis for future growth • Moved from Downtown Brooklyn to DUMBO
foundations: • Infrastructure • Invested in human capital • ‘DevOps’ culture begins in earnest ….. • A lot of reflection and finding an Engineering identity
MySQL for * • It almost doesn’t matter what you choose, just stick to it • ‘If it moves Graph it’ • Graphite, Ganglia, FITB, Weathermap, Nagios, Naglite ….. • Starting to use this data for work/life balance as well as technical/systems reasons
1:1’s as a core mgmt tool • Eng career planning (Reverb) • Accept failures, but not low standards • Developer on-call • Use of A/B testing • Lots of Prototypes • FeatureFlags & Ramp Up Management Ideals Engineering Ideals
development • Focus on technical visibility throughout the org • Developers responsible for code release (start of DevOps) • Member support rotations for all • Work hard at work/life balance & have data to support
silo holdovers (Sprouter gone!) • Non-Standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
more Engineering focused • Chad to CEO • Kellan to CTO • Allspaw to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
Dev & Ops as being integral to culture • Applying our ‘DevOps’ principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often ‘enforced’ with terrible cultural impact • Build a human and effective security organization
growth in hiring, allow easy transfers • Some major changes around product • Increased focus on community • Internationalisation • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp - not just the bottom line
power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data centre usage & partners • ‘Make the world more like Etsy’ - Extending the culture
environment • Allows PCI compliance without disrupting the culture • Interface with the webstack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
term & not just keep the lights on • More headcount than required allows us to take chances • Focus on info exchange, internally & externally with communities • Open source all of the things
and culture • Know when to flick sights from short to longer term goals • Pursue dynamic engineering resource allocation • Do not allow increasing org size to dictate culture
the hard engineering wins taken care of … • Time to focus internally • No engineer can know everything any longer • Need to maintain the culture of transparency & trust • Really was the year of internal tooling to achieve this
and aid postmortems • Moved to Vertica for BI data & metrics • Superbit allowed simple querying of Vertica & big data by anyone who knows SQL • Catapult launched to relate metrics to experiments • Begin a refocus on a Mobile/API First product vision
with tooling that levels access and allows interrogation by ALL • Conscious effort on internal tooling to minimize the pain of large & complex stack • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future
can be accessed, evaluated and contextualized • Have you reached a point where no one can know everything? • While tooling can’t create culture it can help you support it • Be free to apply your culture in new ways • Inward focus cannot lead to outward blindness, tech changes fast
1st day, not just Engineers • Yearly planning is restructured • Take account of a growing Etsy • San Francisco opens as 1st non-Brooklyn Eng hub • Acquire & integrate A Little Market with Etsy
A Little Market acquired • Integrating another engineering culture can be tough • Etsy’s culture is ‘different’ & this can be a big step • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate
to ElasticSearch/Logstash/ Kibana (ELK) • Mobile CI infrastructure embedded & ramped up • API First a huge effort and development push • Mobile First as an increasing product focus • Technical work for quality of life - on-call sleep tracking
lessons • Changing of a core tool require huge comms investment • Without it enclaves & silos can form to resist change • Explain the whys not just in terms of technicals or $$ • Fully understanding all use cases, not just the main ones • Don’t settle for a half complete end goal, go the distance
the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Still use API to enable feature flag driven native apps
✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases!
Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve written per release Imagine that we’ll write
Collect sleep data for on-call • Analyse in a variety of manners • Sleep lost when on-call/pagerduty • Alert on VPN/SSH logins while asleep • Focus on data for quality of life
now M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & comms • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all
growth or M&A activity • It takes effort to keep it so however • Ensure your API is up to the job of supporting Mobile 1st • Ensure core tooling changes are understood & embraced by all • Communicate your Eng culture & history to new hires
continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture • Get smart and use them to test, support and improve it
McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, what went wrong, what went right (Ross Snyder) Slides & Video: http://bit.ly/po8zIj Etsy’s journey to continuous integration for mobile apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean time to sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: http://ryanfrantz.com/mtts/
iodboi
cybozuinsideout
bkuhlmann
ks91
tamaiyutaro
thomasvitale
maroon1st
kkato1
tomu28
ryysud
coconala_engineer
rainerhahnekamp
oracle4engineer
eileencodes
rasmusluckow
jlugia
csswizardry
mthomps
sugarenia
yeseniaperezcruz
mraible
samlambert
tanoku
revolveconf
