Tweet
Tweet

Slide 1

Slide 1 text

Scaling R&D Organization While Maintaining Quality Aviran Mordo Head Of Back-End Engineering @ Wix @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com

Slide 2

Slide 2 text

Wix In Numbers • 44M users from 190 countries • Static storage is >800TB of data • 3 Data centers + 2 Clouds (Google, Amazon) • 700M HTTP requests/day • ~600 people work at Wix • Of which ~ 200 in R&D

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Jan 2014 Deployments (production changes) per month

Slide 5

Slide 5 text

Structures for Scalability • There are 2 key common structures in the industry • Functional unit model • Business unit model

Slide 6

Slide 6 text

Functional Model • Disadvantages • Lack of product ownership • Lack of product level expertise • Hard to predict and plan product roadmap • Cross-function communication is hard • Less focus on delivery and TTM

Slide 7

Slide 7 text

Business Unit Model • Disadvantages • Product integration is hard • Resource and work duplication • Architecture alignment is hard • Technology knowledge sharing is hard • Limited opportunity for professional development

Slide 8

Slide 8 text

• There is no perfect model • It depends on the company’s current challenges, life cycle phase and culture • Every model should be tuned constantly and evolve with the company Our Assumptions

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

• Guild – a group of people that share expertise, knowledge, tools, code, and practices • Gang – a group of people that work in a related products • Compose of all required resources from the different disciplines (Guild) Wix’s Gangs & Guilds Model – How? Guild Gang Guild Gang Gang

Slide 11

Slide 11 text

Wix’s Gangs & Guilds Model – Why? • Technical power of the Guild • Independence of the Gang • Healthy balance between product and tech • Product features and technical equal in priority Guild Leader What How Guild Leader How Team member Team member

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Dev-Centric Culture

Slide 14

Slide 14 text

Traditional Dev Pipeline Product Dev QA Operations 19:14

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Product Dev QA Operations 19:14

Slide 17

Slide 17 text

What Is The Common Denominator? • Product manager • Project manager • QA • Operations • DBA

Slide 18

Slide 18 text

Dev Centric Culture – Involve The Developer – How? • Product definition (with product) • Development (with architect) • Testing (with QA developers) • Deployment / Rollback(with operations) • Monitoring / BI (with BI team) • DevOps – to enable deployment and rollback, fully automated Developer Product QA Management Operation BI

Slide 19

Slide 19 text

esponsibility wnership uality Dev Centric Culture – Why?

Slide 20

Slide 20 text

Project Rotation – How? • Team usually has more than one project • Team members rotate tasks between projects • Developers can switch teams and Gangs

Slide 21

Slide 21 text

Project Rotation – Why? • There is no single person with a knowledge on a specific component • Ongoing review • Keep it interesting • It is everyone's responsibility • Everybody owns everything

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Quality Thursday – How? • A software engineer works 4 days with his Gang • Thursday is a Guild day. • Developers stop working on their designated projects and conduct quality enhancing activities with the Guild.

Slide 24

Slide 24 text

Quality Thursday – Why? • Assimilates people into the Guild • Builds cross-team relationships • Shares knowledge • Enhances the culture • Promotes innovation

Slide 25

Slide 25 text

Bug Hunt – How? • Service owner picks a service running on production • Explains the service to all Guild members • What the service does • Every exception thrown in production • Performance matrix • Get a list of AI from group feedback (sometimes for dependent services)

Slide 26

Slide 26 text

Bug Hunt - Why? • Teach developers about services they don’t own • Understand how your application behaves on production • Open discussion and ideas from group members • Discover unexpected use patterns • Find bugs on production

Slide 27

Slide 27 text

Retrospective – How ? • Whenever developers encounter issues, they are encouraged to post them on a board for public discussion • Anything can be discussed • Topics posted on the board during the week constitute the agenda of the retrospective

Slide 28

Slide 28 text

Retrospective – Why ? • Solve problems • Share lesson learned. • Suggestions on how to improve: • Our team • Quality of products • Process • Effectiveness of the R&D organization.

Slide 29

Slide 29 text

Tech Talks – How? • Every developer can give a talk about anything • People from other departments in the company • Guest speakers • Open to anyone from Wix • Using Meetup http://www.meetup.com/at-wix/ to invite outsiders to our internal talks (if appropriate) • Talks videos http://goo.gl/IDqXTi

Slide 30

Slide 30 text

Tech Talks – Why ? • Training new and old employees • Educating about other activities in the R&D • Educating about other departments activities and concerns in the company

Slide 31

Slide 31 text

Thursday Tasks – How ? • Pay technical debt (preferably not your own) • Work on a task / feature NOT in your project • Read and learn something new • Open source a project • 1 on 1 mentoring

Slide 32

Slide 32 text

Thursday Tasks – Why? • Learn other projects (easier to step in if necessary) • Unbiased review of other projects • Learn about problems and solutions other teams faced and solved that you may also encounter. • Find repeating patterns across projects and generalize a solution • Find interesting projects to switch to. • Leave the code in a better shape than they got it

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Test Driven Development – How? • No new code is pushed to Git without being fully tested • We currently have around 10,000 automated tests • Before fixing a bug first write a test to reproduce the bug • Cover legacy (untested) systems with Integration tests 19:14

Slide 35

Slide 35 text

Test Driven Development – Why? • Developers are responsible for their own code • Less bugs • Easier to enter someone else’s project

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Do Not Compromise On Hiring • Hire only good people • Fit the culture • Excellent technically • Candidates can be dropped • By anyone • At any time • If there is any doubt, then there is no doubt

Slide 38

Slide 38 text

Quality Trust Independence Growth

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Dedicated Resources = Organization Structure

Slide 41

Slide 41 text

Minimize Architectural Dependencies • Service oriented architecture (SOA) • Separation of client (UI) and server projects • Dedicated data stores for each service • Stateless services Lesson learned: System architecture that decouples concerns

Slide 42

Slide 42 text

Communication Channels • To company wide activities • To knowledge centers • To key personnel

Slide 43

Slide 43 text

Quality Trust Independence Growth

Slide 44

Slide 44 text

Lessons Learned: Growing Teams

Slide 45

Slide 45 text

Seeds Of Ambassadors • Train a group of ambassadors that practice dev- centric culture from the Guild • Build new teams with at least one dev-centric ambassador to assimilate new teams into the culture. • Beware of hiring more people than you can train / assimilate successfully

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com Q&A Read some more about maintaining quality R&D: http://goo.gl/c3WLsz