Scaling R&D While Maintaining Quality - ILTeckTalks

Scaling R&D While Maintaining Quality - ILTeckTalks

Case study of how Wix supports rapid growth while maintaining quality by company structure, Games of Gangs - a game like activity that promotes knowledge sharing and more


Aviran Mordo

December 08, 2014


  1. Scaling  R&D  Organiza/on   While  Maintaining  Quality Aviran Mordo Head

    of Back-End Engineering @ Wix @aviranm
  2. Wix  In  Numbers •  53M  users  from  190  countries  

    •  Sta5c  storage  is  >1PB  of  data   •  3  Data  centers  +  2  Clouds  (Google,  Amazon)   •  1.5B  HTTP  requests/day   •  ~800  people  work  at  Wix   •  Of  which  ~  300  in  R&D  
  3. None
  4. May 2014 Deployments  /produc5on  changes  per  month  

  5. Structures  for  Scalability •  There  are  2  key  common  structures

     in  the  industry   •  Func5onal  unit  model   •  Business  unit  model  
  6. Func/onal  Model •  Disadvantages   •  Lack  of  product  ownership

        •  Lack  of  product  level  exper5se     •  Hard  to  predict  and  plan  product   roadmap   •  Cross-­‐func5on  communica5on  is   hard   •  Less  focus  on  delivery  and  TTM  
  7. Business  Unit  Model •  Disadvantages   •  Product  integra5on  is

     hard   •  Resource  and  work  duplica5on   •  Architecture  alignment  is  hard   •  Technology  knowledge  sharing  is   hard   •  Limited  opportunity  for   professional  development  
  8. •  There  is  no  perfect  model     •  It

     depends  on  the  company’s  current  challenges,   life  cycle  phase  and  culture   •  Every  model  should  be  tuned  constantly  and  evolve   with  the  company Our  Assump/ons  
  9. None
  10. •  Guild  –  a  group  of  people   that  share

     exper5se,   knowledge,  tools,  code,   and  prac5ces   •  Gang  –  a  group  of  people   that  work  in  a  related   products   •  Compose  of  all  required   resources  from  the   different  disciplines   (Guild)     Wix’s  Gangs  &  Guilds  Model Guild Gang Guild Gang Gang
  11. Wix’s  Company  Model Guild Gang Guild Gang Gang Company • 

    Company  focus  on  large  segment   •  Has  all  the  resources  it  needs  to   be  independent   •  People  within  the  company  are   aligned  with  the  Guilds     CEO  
  12. Wix’s  Gangs  &  Guilds  Model  –  Why? •  Technical  power

     of  the  Guild   •  Independence  of  the  Gang   •  Healthy  balance  between  product  and  tech   •  Product  features  and  technical  equal  in  priority   Guild  Leader What How Guild  Leader How Team  member Team  member
  13. Quality   Trust   Independence   Growth  

  14. Dev-­‐Centric  Culture

  15. Tradi/onal  Dev  Pipeline Product   Dev   QA   Opera5ons

  16. None
  17. Product   Dev   QA   Opera5ons   15:57  

  18. What  Is  The  Common  Denominator? •  Product  manager   • 

    Project  manager   •  QA   •  Opera5ons   •  DBA  
  19. Dev  Centric  Culture  –  Involve  The  Developer  –  How? • 

    Product  defini5on  (with  product)     •  Development  (with  architect)   •  Tes5ng  (with  QA  developers)   •  Deployment  /  Rollback(with  opera5ons)     •  Monitoring  /  BI  (with  BI  team)   •  DevOps  –  to  enable  deployment  and   rollback,  fully  automated   Developer   Product   QA   Management   Opera5on   BI  
  20. esponsibility   wnership   uality   haring   Dev  Centric

     Culture  –  Why?
  21. None
  22. Project  Rota/on  –  How? •  Team  usually  has  more  than

     one  project   •  Team  members  rotate  tasks  between  projects   •  Developers  can  switch  teams  and  Gangs  
  23. Project  Rota/on  –  Why? •  There  is  no  single  person

     with  a  knowledge  on  a   specific  component   •  Ongoing  review   •  Keep  it  interes5ng   •  It  is  everyone's  responsibility   •  Everybody  owns  everything  
  24. None
  25. Guild  Day •  Engineer  works  4  days  with  his  company

      •  Thursday  is  Guild  day.   •  Developers  conduct  quality  enhancing  ac5vi5es   with  the  Guild.        
  26. Guild  Day  Goals •  Builds  cross-­‐team  rela5onships   •  Shares

     knowledge     •  Assimilate  the  culture   •  Lesson  learned   •  Con5nuous  improvement   •  Promotes  innova5on   •  Professional  development  
  27. Produc/on  Bug  Hunt •  Service  owner  picks  a  service  running

     on  produc5on   •  Present  the  service  to  all  Guild  members   •  What  the  service  does   •  Excep5ons  thrown  in  produc5on   •  Performance  matrix   •  Build  warnings   •  Get  a  list  of  AI  from  group  feedback  (some5mes  for   dependent  services)      
  28.  Produc/on  Bug  Hunt  -­‐  Why? •  Improve  service  stability  

    •  Teach  developers  about  services  they  don’t  own   •  Understand  how  services  behave  on  produc5on   •  Open  discussion  and  ideas  from  group  members   •  Discover  unexpected  use  pajerns   •  Find  bugs  on  produc5on   •  Lessons  learned  from  other  teams  experience  
  29. Retrospec/ve  –  How  ?   •  Whenever  developers  encounter  issues

     /  dilemmas,   they  are  encouraged  to  post  them  on  a  board  (pulse)   for  public  discussion   •  Topics  posted  on  the  board  during  the  week   cons5tute  the  agenda  of  the  retrospec5ve  
  30. None
  31. Retrospec/ve  –  Why  ? •  Main  tool  for  con5nuous  improvement

        •  Share  lesson  learned.   •  Solve  problems     •  Sugges5ons  on  how  to  improve:   •  Our  team   •  Quality  of  products   •  Process     •  Effec5veness  of  the  R&D  organiza5on.    
  32. None
  33. Tech  Talks  –  How? •  Developers  give  a  tech  talks

      •  People  from  other  departments  in  the  company   •  Guest  speakers   •  Open  to  anyone  from  Wix   •  Using  Meetup  hjp://­‐wix/  to   invite  outsiders  to  our  internal  talks  (if  appropriate)   •  Talks  videos  hjp://  
  34. Tech  Talks  –  Why  ? •  Training  employees   • 

    Knowledge  sharing   •  Educa5ng  about  other  ac5vi5es  at  Wix   •  Professional  development    
  35. Thursday  Tasks

  36. Games  of  Gangs •  Build  tools  that  help  developers  

    •  Enhance  plalorm  /  framework   •  Pay  technical  (legacy)  debt  (refactor,  etc’)   •  Improve  tests   •  Research  something  new  for  your  “company”   •  Work  on  a  task  for  a  different  “company”   •  Open  source  a  project   •  1  on  1  mentoring  
  37. None
  38. Games  of  Gangs  –  Why? •  Improve  code  quality  

    •  Improve  developer  produc5vity   •  Find  repea5ng  pajerns  across  projects  -­‐  generalize  a   solu5on  or  improve  the  infrastructure   •  Learn  other  projects  (easier  to  step  in  if  necessary)   •  Unbiased  review  of  other  projects   •  Learn  about  problems  and  solu5ons  other  teams   faced  and  solved  that  you  may  also  encounter.   •  It  fun  and  breaks  the  day  to  day  rou5ne    
  39. None
  40. Test  Driven  Development  –  How? •  No  new  code  is

     pushed  to  Git  without  being  fully  tested   •  We  currently  have  around  10,000  automated  tests   •  Before  fixing  a  bug  first  write  a  test  to  reproduce  the  bug   •  Cover  legacy  (untested)  systems  with  Integra5on  tests   15:57  
  41. Test  Driven  Development  –  Why? •  Developers  are  responsible  for

     their  own  code   •  Less  bugs   •  Easier  to  enter  someone  else’s  project  
  42. None
  43. Do  Not  Compromise  On  Hiring •  Hire  only  good  people

      •  Fit  the  culture   •  Excellent  technically   •  Candidates  can  be  dropped   •  By  anyone   •  At  any  5me   •  If  there  is  any  doubt,  then   there  is  no  doubt  
  44. Quality   Trust   Independence   Growth  

  45. None
  46. Dedicated  Resources  =  Organiza/on  Structure

  47. Minimize  Architectural  Dependencies •  Service  oriented  architecture  (SOA)   • 

    Separa5on  of  client  (UI)  and  server  projects   •  Dedicated  data  stores  for  each  service   •  Stateless  services   Lesson  learned:  System  architecture  that  decouples  concerns  
  48. Communica/on  Channels •  To  company  wide  ac5vi5es   •  To

     knowledge  centers   •  To  key  personnel  
  49. Quality   Trust   Independence   Growth  

  50. Lessons  Learned:  Growing  Teams

  51. Seeds  Of  Ambassadors •  Train  a  group  of  ambassadors  that

     prac5ce  dev-­‐ centric  culture  from  the  Guild   •  Build  new  teams  with  at  least  one  dev-­‐centric   ambassador  to  assimilate  new  teams  into  the   culture.   •  Beware  of  hiring  more  people  than  you  can  train  /   assimilate  successfully  
  52. None
  53. CFO  asks  CEO:  “What  happens  if  we   invest  in

     developing  our  people  and  they   leave  us?”     CEO:  “What  happened  if  we  don’t  and   they  stay?”  
  54. Yes!  We  Are  Hiring  –  

  55. Aviran Mordo Head of Back-End Engineering @ Wix @aviranm Q&A Read  some  more  about  maintaining  quality  R&D:  hjp://