Share
Embed
Slide 1
Slide 1 text
Development Process
Refactoring Case
Study
Kazuki Matsuda / LINE
Slide 2
Slide 2 text
Abstract
› In this session, we will discuss the process improvements
of server-side developer team.
› We focused on internal development processes, especially repository
management and review. Because we believe that quick feedback and
the continuous small improvements is the basis of improving throughput.
› As a result, we saved 89% of time until PR is merged,
and many qualitative surveys showed an increase in development speed.
Slide 3
Slide 3 text
We are //manager.line.biz dev
› CMS of LINE Official Account.
.
Slide 4
Slide 4 text
We are //manager.line.biz dev
› CMS of LINE Official Account.
› Manage audience.
› Create contents /
Broadcast message.
› Check analytics.
› Billing.
› Internal data distribution.
.
Slide 5
Slide 5 text
We are //manager.line.biz dev
› CMS of LINE Official Account.
› Manage audience.
› Create contents /
Broadcast message.
› Check analytics.
› Billing.
› Internal data distribution.
› Composed by
› 20+ microservices by our team
› Uncountable microservices by others
› 300+ VMs. (& PMs.)
Slide 6
Slide 6 text
Our Team Members
Units Over One Team
3
Merged PR / Month
248
Core Developer
13
Slide 7
Slide 7 text
Agenda › Types of our refactoring.
› The refactoring case we did.
› Summary.
Slide 8
Slide 8 text
Types of Refactoring
Slide 9
Slide 9 text
Types of refactoring
Development Process Refactoring
Changing rule for development process.
= A.k.a. BPR (Business Process Re-engineering)
Code Refactoring
Rewrite your code without functional change.
For maintainability, runtime optimization etc.
Slide 10
Slide 10 text
Cases & Solutions
Process Refactoring Case Study
Slide 11
Slide 11 text
Our process refactoring point.
Git Repository Size
Microservices but
Mono-repository.
Branch Management
Simple Branch Strategy.
Code Reviews
Reviews first.
Slide 12
Slide 12 text
Git Repository Size
Slide 13
Slide 13 text
Case Study: Git Repository Size
› Before: Micro-repositories per microservices / modules.
› For explicit R&R of module and developer.
› After: Single (Monolithic) repository for all team member
with directories for each micro-services.
› Why?
Slide 14
Slide 14 text
Micro-repository Problem
› Problem:
› Many PRs and reviews needed
for single feature.
› Dependency version, coding
styles, CI tools not aligned.
› All of those is a result of “Micro-
repository (Anti) Pattern”.
Domain
Internal API
manager.line.biz
PR
PR PR
Slide 15
Slide 15 text
Case Study: Split module or not?
› Before: Micro-repository mind
› I think we should split modules for new feature.
› But time and individual efforts needed.
› After: Mono-repository mind.
› Just create directory. Moving files. Done. With history.
› Module test coverage ++.
Slide 16
Slide 16 text
Conclusion: Repository Management
› Mono repository is very efficient way to development.
› Applied in our team. (13Dev, 3 Units
› Ref:
› Big IT company also uses mono-repo. ※
※ Rachel Potvin and Josh Levenberg, “Why Google stores billions of lines of code in a single repository,” Communications of the ACM, 59 No. 7 (2016): 78-87.
Slide 17
Slide 17 text
Branch Management
Slide 18
Slide 18 text
Case Study: Branch Problem
› “@dev Internal test environment isn’t working.
Can someone please check it?”
› Problem: Even many developers coding on this repo.
But no one noticed test environment is broken.
› Because: No one respect (coding on) development branch.
Slide 19
Slide 19 text
Anti-Pattern: Development branch
Mainline (A.K.A master) / Developers on it
Development (A.K.A. Beta)
QA, Planner, other teams on it
New features
No problem for small team.
But uncontrollable when service scaled.
Slide 20
Slide 20 text
Anti-Pattern: Development branch
› Sandbox branch may work well in small repository.
› But not for large repository.
Mainline (A.K.A master) / Developers on it
Development (A.K.A. Beta)
QA, Planner, other teams on it
New features
No problem for small team.
But uncontrollable when service scaled.
Slide 21
Slide 21 text
Case Study: Branch Problem
› Distributed version control is too difficult for human.
› Especially team management.
› Anti-Pattern: THEIR(Team’s) latest is not MY latest.
Slide 22
Slide 22 text
Case Study: Branch Problem
Solution
› Distributed version control is too difficult for human.
› Especially team management.
› Anti-Pattern: THEIR(Team’s) latest is not MY latest.
› Solution: Define single truth of mainline.
› Starting point of development
= reviewed = internal beta = next release candidate.
Slide 23
Slide 23 text
Repository
& Branch
Summary
› Stop create multi repository.
› Use directory instead.
› Stop creating multi latest version.
› Make feature branch lifetime short.
› DO NOT SPLIT.
1. New feature starting point.
2. Merge destination.
3. Internal test env.
4. Release (Release Candidate) env.
Slide 24
Slide 24 text
Reviews
Slide 25
Slide 25 text
Reviews
› In our case, code review is bottleneck.
› Bottleneck = Upper limit of your team throughput.
› Improvements or cost cutting not related to the bottleneck
does not increase throughput.
› Bottleneck but no one prioritize it.
Slide 26
Slide 26 text
Reviews: Checklist.
› Principle.
› Reviews first.
› Checklist
› Do you finish code reviews before starting write your code?
› Do you subscribe notification when code review requested?
› Do you check PR list regularly like checking slack?
Slide 27
Slide 27 text
Reviews
› Question: Pending PR is not important because they finally
releases. My interruption cost is higher than benefit of fast feedback
for peers. Isn’t it?
› NO!
› Pending PR is stock.
› Stock is not assets. It’s liabilities.
› Of course, you need a “zone”.
Please remind reviewing when you interrupted.
Slide 28
Slide 28 text
Result: Culture and trust of feedback
0
20
40
60
80
100
120
2018/04
2018/05
2018/06
2018/07
2018/08
2018/09
2018/10
2018/11
2018/12
2019/01
2019/02
2019/03
2019/04
2019/05
2019/06
2019/07
2019/08
2019/09
2019/10
2019/11
2019/12
2020/01
2020/02
2020/03
2020/04
2020/05
2020/06
2020/07
2020/08
2020/09
2020/10
Review Duration (Merged – Created) in Hours, Median
Hours
Slide 29
Slide 29 text
Result: Culture and trust of feedback
0
20
40
60
80
100
120
2018/04
2018/05
2018/06
2018/07
2018/08
2018/09
2018/10
2018/11
2018/12
2019/01
2019/02
2019/03
2019/04
2019/05
2019/06
2019/07
2019/08
2019/09
2019/10
2019/11
2019/12
2020/01
2020/02
2020/03
2020/04
2020/05
2020/06
2020/07
2020/08
2020/09
2020/10
Review Duration (Merged – Created) in Hours, Median
Things getting worse
2019: 26.1 Hours
Hours
Slide 30
Slide 30 text
Result: Culture and trust of feedback
Most important thing is the culture and trust that
submitting review is one of the fastest way to got feedback and improve your ideas.
0
20
40
60
80
100
120
2018/04
2018/05
2018/06
2018/07
2018/08
2018/09
2018/10
2018/11
2018/12
2019/01
2019/02
2019/03
2019/04
2019/05
2019/06
2019/07
2019/08
2019/09
2019/10
2019/11
2019/12
2020/01
2020/02
2020/03
2020/04
2020/05
2020/06
2020/07
2020/08
2020/09
2020/10
Review Duration (Merged – Created) in Hours, Median
Things getting worse Review First
2019: 26.1 Hours
2020: 2.9 Hours
Hours
Slide 31
Slide 31 text
Review
Summary
› Always reviews first.
› Minimize team’s pending PR.
› Don’t maximize only your efficiency.
Respect your team throughput.
Slide 32
Slide 32 text
Summary
Slide 33
Slide 33 text
Summary
› Two types of refactoring.
› Process Refactoring
› Repository management.
› Branch management.
› Reviews.
› Introduce developer’s voice.
Slide 34
Slide 34 text
Two types of refactoring
› 2 types of refactoring.
› Process refactoring.
› ≒ BPR (Business Process Re-engineering).
› Code refactoring. (Design patterns etc.)
› Both important.
But fast code refactoring depends on process refactoring.
Slide 35
Slide 35 text
Repositories / Branches
Make everything simple for scalable team.
› Making repository simple.
› Use directory instead multi-micro-repository.
› Making branch strategy simple.
› Only one special mainline branch (main / master / mainline).
Slide 36
Slide 36 text
Reviews
For culture and trust
› Principle.
› Reviews first.
› Checklist
› Do you finish code reviews before starting write your code?
› Do you subscribe notification when code review requested?
› Do you check PR list regularly like checking slack?
› Most important thing is the culture and trust that submitting review
is one of the fastest way to got feedback and improve your ideas.
Slide 37
Slide 37 text
Questionnaire Result
Slide 38
Slide 38 text
Questionnaire Result
› Pros
› Fast release based on fast review. That makes schedule clear.
› Conflict reduced. Many merge operation is not needed.
› Context switching between coding and reviewing is not needed.
Slide 39
Slide 39 text
Questionnaire Result
› Pros
› Fast release based on fast review. That makes schedule clear.
› Conflict reduced. Many merge operation is not needed.
› Context switching between coding and reviewing is not needed.
› Cons
› Need to protect unreleased feature by feature flag.
› But better than managed by branch.
Slide 40
Slide 40 text
Thank you