Share
Embed
Continuous Culture
Culture is always evolving and doesn't come for free. This is the story of Etsy's journey from 4-person startup to an international organization and the culture that we all work to foster.
RyanFrantz
January 14, 2015
Transcript
59 slides
-
@Ryan_Frantz #me • Ryan Frantz • Senior Operations Engineer at
Etsy (Remote) • Built and managed teams in the past • Contributes to and helps maintain culture at Etsy -
@Ryan_Frantz By the Numbers (FY 2013) • Gross Marketplace Sales
(GMS) $1.35 Billion • 44 million members, 1.2 million active sellers • 26 million active listings • Nearly Every Country Performing Transactions • > 600 Employees -
@Ryan_Frantz There WILL be a test… • Culture constantly develops
• Communication is paramount • No divide between managers and engineers • Technology/product can attract people, but culture keeps them -
@Ryan_Frantz Core Engineering Principles • Just Ship • Every engineer
can push to prod at any time • Favor simplicity and speed to ship • Maximize learning • ‘If it moves, graph it’ -
@Ryan_Frantz Continuous Culture v1 • Mike Rembetsy & Patrick McDonnell
• Presented at Velocity Conference • Etsy’s engineering culture evolution 2006-2012 • Slides: http://slidesha.re/1xYxZrG • Video: http://vimeo.com/51310058 -
@Ryan_Frantz 2006 - 2008 Silos and Barriers • 4-person startup
grows to employ 30 - 35 (including 15 ENG) • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases -
@Ryan_Frantz Management Changes • Maria Thomas from NPR promoted to
CEO • Brings a clear understanding that community is very important • Prioritizes a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • “This siloed culture cannot work, we need to start over” -
@Ryan_Frantz 2006-2008 Takeaways • Downtime was an accepted fact of
life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just! -
@Ryan_Frantz 2009 Internal Improvements • As teams grow, big efforts
in good communication • Daily standups • DevTools team • Deployinator • lighttpd -> Apache • Network solidified -
@Ryan_Frantz 2009 Takeaways • Built solid foundations • Invested in
human capital • Strong collaboration • Infrastructure • A lot of reflection and finding an Engineering identity -
@Ryan_Frantz 2010 Standardization & Graphs • Moved to PHP &
MySQL for everything • ‘If it moves, graph it’ • Ganglia, Graphite, FITB, Nagios, Naglite • Starting to use this data for work/life balance as well as technical/systems reasons -
@Ryan_Frantz 2010 Ideals • Blameless Postmortems • 1:1s as a
core mgmt tool • Engineering career planning • Developer on-call • Use of A/B testing • Feature Flags & Ramp Up • Schema Change Thursday Management Ideals Engineering Ideals -
@Ryan_Frantz 2010 Takeaways • Reduce number of technologies used in
development • Focus on technical visibility throughout the org • Developers responsible for code deploy (and on-call!) • Work/life balance focus -
@Ryan_Frantz 2011 Tech Highlights • End of long tail legacy
silo holdovers (Sprouter gone!) • Non-standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software -
@Ryan_Frantz 2011 - Organizational Changes • Senior management to become
more Engineering focused • Chad to CEO • Kellan promoted to CTO • Allspaw promoted to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability -
@Ryan_Frantz 2011 Takeaways • Year of open sourced tools! •
Statsd, Logster, Deployinator, Supergrep, Schemanator • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef) • Security front and center • Security without negative impact to culture? -
@Ryan_Frantz 2011-2012 - A Focus on Security • Security alongside
Dev & Ops as being integral to culture • Applying our core principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often enforced with terrible cultural impact • Build a human and effective security organization -
@Ryan_Frantz 2012 - Growth + Foster Our Values • Explosive
growth in hiring, allow easy team transfers • Some major changes around product • Internationalization • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp -
@Ryan_Frantz What’s a B-Corp ? • Aim to use the
power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data center usage & partners • ‘Make the world more like Etsy’ - Extending the culture -
@Ryan_Frantz 2012 - Technical Achievements • Create wholly separate Payments
environment • Allows PCI compliance without disrupting the culture • Interface with the web stack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers -
@Ryan_Frantz 2012 Takeaways • Do what’s needed to sustain long
term goals & not just keep the lights on • More headcount allows us to take chances • Focus on social/environmental impact, internally & externally with communities • Open source all of the things -
@Ryan_Frantz 2013 - An Interesting Year! • Had many of
the hard engineering wins taken care of • Time to focus internally • No engineer can know everything any longer • Internal tooling • Designated Ops -
@Ryan_Frantz 2013 - Tooling • Morgue: Stores postmortem details (including
IRC logs, images) • Opsweekly: Categorize and report Nagios alerts • Superbit: Allows simple querying of Vertica, Elasticsearch & big data by anyone who knows SQL • Catapult: Communicates experiments and related metrics (including conversion impact) • Begin a refocus on a Mobile/API First product vision -
@Ryan_Frantz 2013 - Designated Ops • Visibility of ongoing projects
• Spreading knowledge • Advocating for engineers • Assignments rotate annually -
@Ryan_Frantz 2013 Takeaways • Democratization of data is made easier
with tooling that levels access and allows interrogation by ALL • Designated Ops • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future -
@Ryan_Frantz 2013 End of Year Snapshot GMS Employees Engineers $1.35
Billion 615 >33% -
@Ryan_Frantz 2014 - Organizational changes • Everyone pushes on their
1st day • Yearly planning is restructured • San Francisco office opens • Acquire & integrate A Little Market -
@Ryan_Frantz Cultural Acquisition • Paris-based A Little Market (ALM) •
Integrating another engineering culture can be tough • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate -
@Ryan_Frantz 2014 - Technical • Migrate from Splunk to ELK
(Elasticsearch/Logstash/Kibana) • Mobile First has increasing product focus • Mobile CI infrastructure embedded & ramped up • API First is huge effort and development push • Technical work for quality of life: On-call sleep tracking -
@Ryan_Frantz ELK Lessons • Changing a core tool requires huge
communication investment • Understanding usage patterns • Accommodating that usage and factoring into the project • Describing the purpose of the project (not just about $$) • Technology can find its way into unexpected corners of your stack! -
@Ryan_Frantz Mobile First • Applying your principles and culture to
the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Use API to enable feature flag-driven native apps -
Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓
✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ? -
@Ryan_Frantz API First • Supporting the Mobile First push &
diversity of clients • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging -
@Ryan_Frantz Why This Approach? • Continuous integration, Continuous Delivery •
Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases! -
Single release Many releases 50K LOC/month Few opportunities for failure
Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve written per release Imagine that we’ll write -
@Ryan_Frantz Sleep Tracking • Experiment with Jawbone UPs & Ops
• Collect on-call sleep data • Analyze sleep lost when on-call • Focus on data for quality of life -
@Ryan_Frantz 2014 Takeaways • Another year of big growth +
M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & communication • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all -
@Ryan_Frantz 2015 Action Items • Culture is still king despite
growth or M&A activity • It takes effort to keep it so however • Ensure our API is up to the job of supporting Mobile First • Ensure core tooling changes are understood & embraced by all • Communicate our engineering culture & history to new hires -
@Ryan_Frantz Conclusions • Culture doesn’t come for free, it takes
continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture -
@Ryan_Frantz Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick
McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, What Went Wrong, What Went Right (Ross Snyder) Slides & video: http://bit.ly/po8zIj Etsy’s Journey to Continuous Integration for Mobile Apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean Time to Sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: http://ryanfrantz.com/mtts/