Scaling R&D Org While Maintaining Quality

As a fast growing company Wix R&D doubles every year. In this talk I will describe how we structured our R&D division, what we are doing to build and keep an "A" team of developers and our dev centric and quality based culture that supports innovation.

Transcript

1. Scaling R&D Organization While Maintaining Quality Aviran Mordo Head Of Back-End Engineering @ Wix @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com
2. Wix In Numbers • 44M users from 190 countries • Static storage is >800TB of data • 3 Data centers + 2 Clouds (Google, Amazon) • 700M HTTP requests/day • ~600 people work at Wix • Of which ~ 200 in R&D
4. Jan 2014 Deployments (production changes) per month
5. Structures for Scalability • There are 2 key common structures in the industry • Functional unit model • Business unit model
6. Functional Model • Disadvantages • Lack of product ownership • Lack of product level expertise • Hard to predict and plan product roadmap • Cross-function communication is hard • Less focus on delivery and TTM
7. Business Unit Model • Disadvantages • Product integration is hard • Resource and work duplication • Architecture alignment is hard • Technology knowledge sharing is hard • Limited opportunity for professional development
8. • There is no perfect model • It depends on the company’s current challenges, life cycle phase and culture • Every model should be tuned constantly and evolve with the company Our Assumptions
10. • Guild – a group of people that share expertise, knowledge, tools, code, and practices • Gang – a group of people that work in a related products • Compose of all required resources from the different disciplines (Guild) Wix’s Gangs & Guilds Model – How? Guild Gang Guild Gang Gang
11. Wix’s Gangs & Guilds Model – Why? • Technical power of the Guild • Independence of the Gang • Healthy balance between product and tech • Product features and technical equal in priority Guild Leader What How Guild Leader How Team member Team member
13. Dev-Centric Culture
14. Traditional Dev Pipeline Product Dev QA Operations 19:14
16. Product Dev QA Operations 19:14
17. What Is The Common Denominator? • Product manager • Project manager • QA • Operations • DBA
18. Dev Centric Culture – Involve The Developer – How? • Product definition (with product) • Development (with architect) • Testing (with QA developers) • Deployment / Rollback(with operations) • Monitoring / BI (with BI team) • DevOps – to enable deployment and rollback, fully automated Developer Product QA Management Operation BI
19. esponsibility wnership uality Dev Centric Culture – Why?
20. Project Rotation – How? • Team usually has more than one project • Team members rotate tasks between projects • Developers can switch teams and Gangs
21. Project Rotation – Why? • There is no single person with a knowledge on a specific component • Ongoing review • Keep it interesting • It is everyone's responsibility • Everybody owns everything
23. Quality Thursday – How? • A software engineer works 4 days with his Gang • Thursday is a Guild day. • Developers stop working on their designated projects and conduct quality enhancing activities with the Guild.
24. Quality Thursday – Why? • Assimilates people into the Guild • Builds cross-team relationships • Shares knowledge • Enhances the culture • Promotes innovation
25. Bug Hunt – How? • Service owner picks a service running on production • Explains the service to all Guild members • What the service does • Every exception thrown in production • Performance matrix • Get a list of AI from group feedback (sometimes for dependent services)
26. Bug Hunt - Why? • Teach developers about services they don’t own • Understand how your application behaves on production • Open discussion and ideas from group members • Discover unexpected use patterns • Find bugs on production
27. Retrospective – How ? • Whenever developers encounter issues, they are encouraged to post them on a board for public discussion • Anything can be discussed • Topics posted on the board during the week constitute the agenda of the retrospective
28. Retrospective – Why ? • Solve problems • Share lesson learned. • Suggestions on how to improve: • Our team • Quality of products • Process • Effectiveness of the R&D organization.
29. Tech Talks – How? • Every developer can give a talk about anything • People from other departments in the company • Guest speakers • Open to anyone from Wix • Using Meetup http://www.meetup.com/at-wix/ to invite outsiders to our internal talks (if appropriate) • Talks videos http://goo.gl/IDqXTi
30. Tech Talks – Why ? • Training new and old employees • Educating about other activities in the R&D • Educating about other departments activities and concerns in the company
31. Thursday Tasks – How ? • Pay technical debt (preferably not your own) • Work on a task / feature NOT in your project • Read and learn something new • Open source a project • 1 on 1 mentoring
32. Thursday Tasks – Why? • Learn other projects (easier to step in if necessary) • Unbiased review of other projects • Learn about problems and solutions other teams faced and solved that you may also encounter. • Find repeating patterns across projects and generalize a solution • Find interesting projects to switch to. • Leave the code in a better shape than they got it
34. Test Driven Development – How? • No new code is pushed to Git without being fully tested • We currently have around 10,000 automated tests • Before fixing a bug first write a test to reproduce the bug • Cover legacy (untested) systems with Integration tests 19:14
35. Test Driven Development – Why? • Developers are responsible for their own code • Less bugs • Easier to enter someone else’s project
37. Do Not Compromise On Hiring • Hire only good people • Fit the culture • Excellent technically • Candidates can be dropped • By anyone • At any time • If there is any doubt, then there is no doubt
38. Quality Trust Independence Growth
40. Dedicated Resources = Organization Structure
41. Minimize Architectural Dependencies • Service oriented architecture (SOA) • Separation of client (UI) and server projects • Dedicated data stores for each service • Stateless services Lesson learned: System architecture that decouples concerns
42. Communication Channels • To company wide activities • To knowledge centers • To key personnel
43. Quality Trust Independence Growth
44. Lessons Learned: Growing Teams
45. Seeds Of Ambassadors • Train a group of ambassadors that practice dev- centric culture from the Guild • Build new teams with at least one dev-centric ambassador to assimilate new teams into the culture. • Beware of hiring more people than you can train / assimilate successfully
47. Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com Q&A Read some more about maintaining quality R&D: http://goo.gl/c3WLsz