Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development Process Refactoring Case Study

Development Process Refactoring Case Study

Kazuki Matsuda
LINE OA Dev Team1 - ANTIFRAGILE TF ANTIFRAGILE TF Lead
https://linedevday.linecorp.com/2020/ja/sessions/6992
https://linedevday.linecorp.com/2020/en/sessions/6992

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 26, 2020
Tweet

Transcript

  1. Development Process Refactoring Case Study Kazuki Matsuda / LINE

  2. Abstract › In this session, we will discuss the process

    improvements of server-side developer team. › We focused on internal development processes, especially repository management and review. Because we believe that quick feedback and the continuous small improvements is the basis of improving throughput. › As a result, we saved 89% of time until PR is merged, and many qualitative surveys showed an increase in development speed.
  3. We are //manager.line.biz dev › CMS of LINE Official Account.

    .
  4. We are //manager.line.biz dev › CMS of LINE Official Account.

    › Manage audience. › Create contents / Broadcast message. › Check analytics. › Billing. › Internal data distribution. .
  5. We are //manager.line.biz dev › CMS of LINE Official Account.

    › Manage audience. › Create contents / Broadcast message. › Check analytics. › Billing. › Internal data distribution. › Composed by › 20+ microservices by our team › Uncountable microservices by others › 300+ VMs. (& PMs.)
  6. Our Team Members Units Over One Team 3 Merged PR

    / Month 248 Core Developer 13
  7. Agenda › Types of our refactoring. › The refactoring case

    we did. › Summary.
  8. Types of Refactoring

  9. Types of refactoring Development Process Refactoring Changing rule for development

    process. = A.k.a. BPR (Business Process Re-engineering) Code Refactoring Rewrite your code without functional change. For maintainability, runtime optimization etc.
  10. Cases & Solutions Process Refactoring Case Study

  11. Our process refactoring point. Git Repository Size Microservices but Mono-repository.

    Branch Management Simple Branch Strategy. Code Reviews Reviews first.
  12. Git Repository Size

  13. Case Study: Git Repository Size › Before: Micro-repositories per microservices

    / modules. › For explicit R&R of module and developer. › After: Single (Monolithic) repository for all team member with directories for each micro-services. › Why?
  14. Micro-repository Problem › Problem: › Many PRs and reviews needed

    for single feature. › Dependency version, coding styles, CI tools not aligned. › All of those is a result of “Micro- repository (Anti) Pattern”. Domain Internal API manager.line.biz PR PR PR
  15. Case Study: Split module or not? › Before: Micro-repository mind

    › I think we should split modules for new feature. › But time and individual efforts needed. › After: Mono-repository mind. › Just create directory. Moving files. Done. With history. › Module test coverage ++.
  16. Conclusion: Repository Management › Mono repository is very efficient way

    to development. › Applied in our team. (13Dev, 3 Units › Ref: › Big IT company also uses mono-repo. ※ ※ Rachel Potvin and Josh Levenberg, “Why Google stores billions of lines of code in a single repository,” Communications of the ACM, 59 No. 7 (2016): 78-87.
  17. Branch Management

  18. Case Study: Branch Problem › “@dev Internal test environment isn’t

    working. Can someone please check it?” › Problem: Even many developers coding on this repo. But no one noticed test environment is broken. › Because: No one respect (coding on) development branch.
  19. Anti-Pattern: Development branch Mainline (A.K.A master) / Developers on it

    Development (A.K.A. Beta) QA, Planner, other teams on it New features No problem for small team. But uncontrollable when service scaled.
  20. Anti-Pattern: Development branch › Sandbox branch may work well in

    small repository. › But not for large repository. Mainline (A.K.A master) / Developers on it Development (A.K.A. Beta) QA, Planner, other teams on it New features No problem for small team. But uncontrollable when service scaled.
  21. Case Study: Branch Problem › Distributed version control is too

    difficult for human. › Especially team management. › Anti-Pattern: THEIR(Team’s) latest is not MY latest.
  22. Case Study: Branch Problem Solution › Distributed version control is

    too difficult for human. › Especially team management. › Anti-Pattern: THEIR(Team’s) latest is not MY latest. › Solution: Define single truth of mainline. › Starting point of development = reviewed = internal beta = next release candidate.
  23. Repository & Branch Summary › Stop create multi repository. ›

    Use directory instead. › Stop creating multi latest version. › Make feature branch lifetime short. › DO NOT SPLIT. 1. New feature starting point. 2. Merge destination. 3. Internal test env. 4. Release (Release Candidate) env.
  24. Reviews

  25. Reviews › In our case, code review is bottleneck. ›

    Bottleneck = Upper limit of your team throughput. › Improvements or cost cutting not related to the bottleneck does not increase throughput. › Bottleneck but no one prioritize it.
  26. Reviews: Checklist. › Principle. › Reviews first. › Checklist ›

    Do you finish code reviews before starting write your code? › Do you subscribe notification when code review requested? › Do you check PR list regularly like checking slack?
  27. Reviews › Question: Pending PR is not important because they

    finally releases. My interruption cost is higher than benefit of fast feedback for peers. Isn’t it? › NO! › Pending PR is stock. › Stock is not assets. It’s liabilities. › Of course, you need a “zone”. Please remind reviewing when you interrupted.
  28. Result: Culture and trust of feedback 0 20 40 60

    80 100 120 2018/04 2018/05 2018/06 2018/07 2018/08 2018/09 2018/10 2018/11 2018/12 2019/01 2019/02 2019/03 2019/04 2019/05 2019/06 2019/07 2019/08 2019/09 2019/10 2019/11 2019/12 2020/01 2020/02 2020/03 2020/04 2020/05 2020/06 2020/07 2020/08 2020/09 2020/10 Review Duration (Merged – Created) in Hours, Median Hours
  29. Result: Culture and trust of feedback 0 20 40 60

    80 100 120 2018/04 2018/05 2018/06 2018/07 2018/08 2018/09 2018/10 2018/11 2018/12 2019/01 2019/02 2019/03 2019/04 2019/05 2019/06 2019/07 2019/08 2019/09 2019/10 2019/11 2019/12 2020/01 2020/02 2020/03 2020/04 2020/05 2020/06 2020/07 2020/08 2020/09 2020/10 Review Duration (Merged – Created) in Hours, Median Things getting worse 2019: 26.1 Hours Hours
  30. Result: Culture and trust of feedback Most important thing is

    the culture and trust that submitting review is one of the fastest way to got feedback and improve your ideas. 0 20 40 60 80 100 120 2018/04 2018/05 2018/06 2018/07 2018/08 2018/09 2018/10 2018/11 2018/12 2019/01 2019/02 2019/03 2019/04 2019/05 2019/06 2019/07 2019/08 2019/09 2019/10 2019/11 2019/12 2020/01 2020/02 2020/03 2020/04 2020/05 2020/06 2020/07 2020/08 2020/09 2020/10 Review Duration (Merged – Created) in Hours, Median Things getting worse Review First 2019: 26.1 Hours 2020: 2.9 Hours Hours
  31. Review Summary › Always reviews first. › Minimize team’s pending

    PR. › Don’t maximize only your efficiency. Respect your team throughput.
  32. Summary

  33. Summary › Two types of refactoring. › Process Refactoring ›

    Repository management. › Branch management. › Reviews. › Introduce developer’s voice.
  34. Two types of refactoring › 2 types of refactoring. ›

    Process refactoring. › ≒ BPR (Business Process Re-engineering). › Code refactoring. (Design patterns etc.) › Both important. But fast code refactoring depends on process refactoring.
  35. Repositories / Branches Make everything simple for scalable team. ›

    Making repository simple. › Use directory instead multi-micro-repository. › Making branch strategy simple. › Only one special mainline branch (main / master / mainline).
  36. Reviews For culture and trust › Principle. › Reviews first.

    › Checklist › Do you finish code reviews before starting write your code? › Do you subscribe notification when code review requested? › Do you check PR list regularly like checking slack? › Most important thing is the culture and trust that submitting review is one of the fastest way to got feedback and improve your ideas.
  37. Questionnaire Result

  38. Questionnaire Result › Pros › Fast release based on fast

    review. That makes schedule clear. › Conflict reduced. Many merge operation is not needed. › Context switching between coding and reviewing is not needed.
  39. Questionnaire Result › Pros › Fast release based on fast

    review. That makes schedule clear. › Conflict reduced. Many merge operation is not needed. › Context switching between coding and reviewing is not needed. › Cons › Need to protect unreleased feature by feature flag. › But better than managed by branch.
  40. Thank you