Story of Rewriting 8-Year-Old Perl Product with Kotlin

About Me Kohei Ohara - Joined LINE in 2018 -
Server-side engineer of LINE Point Club - Writing code in Perl, Kotlin, and Java

Agenda - About LINE Point Club - Why did we
decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

LINE Point Club

- You can earn ”LINE Point” Earn Tab

- You can exchange ”LINE Point” Use Tab

How much traffic Req / Sec - Avg 700 Req
/ Day 57M Req / Sec - Peak 15K

System overview in Perl era cv-worker cv-worker apps for Use
Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer

Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06

Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06 Launched in 2015-04

Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Difficulties of having 2 different language apps • Allocation of engineering resources L • Develop the same features in different languages L

Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer I’ll explain more

System overview in Perl era (Earn Tab) media admin cv-worker
Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway

Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Amon2 (Perl)

Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Spring Boot (Kotlin/Java)

Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway We cannot use HBase from Perl directly L

Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway

Why Kotlin? - Most Point Club apps other than Perl
are written in Kotlin Why not Java? Java/Kotlin is most used language in the company - Engineer resource J - There is a Java/Kotlin ecosystem in the company J

Why did we decide to rewrite Perl to Kotlin? -
Complicated system architecture L - Earn tab is implemented in Perl, use tab is implemented in Kotlin - HBase gateway app for Perl - Complicated features L - Many features that are no longer in use - Due to the small number of Perl engineers, it seemed impossible to continue development in the future L - Before the start of the project, there seemed to be no major changes for the next six months J

Why did we decide to rewrite Perl to Kotlin? -
The number of people maintaining Perl libraries is decreasing - Perl‘s client libraries for recent middleware are not rich - HBase - We have tried to use HBase from Perl several times - But we couldn't get it to work as we'd hoped - We ended up implementing HBase gateway in Java/Kotlin - Redis Cluster

System overview after remaking media core admin cv-worker Kafka Redis
Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Spring Boot (Kotlin)

Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Same DB as old system

System overview: gRPC media core admin cv-worker Kafka Redis Clusters
Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

System overview: Kafka as job queue media core admin cv-worker
Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

System overview: HBase media core admin cv-worker Kafka Redis Clusters

System overview: HBase media core admin cv-worker Kafka Redis Clusters
Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL HBase gateway is no longer needed J

System overview: admin-app media core admin cv-worker Kafka Redis Clusters

System overview: admin-app media core cv-worker Kafka Redis Clusters Redis
gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin

System overview: media-app media core admin cv-worker Kafka Redis Clusters

System overview: media-app media core admin cv-worker Kafka Redis Clusters
Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL media-app can use the data from both the earn and use tabs J

Remake project developers 3 Java/Kotlin engineers - They have a
lot of experience in developing for Java/Kotlin and Spring Boot - They don’t know much about the specification of earn tab in the Perl era - In the early days of the project, they were working on the base of the development - Without them, we could not have completed the Remake Project J 3 Perl engineers - They know about the specifications of the earn tab - In the early days of the project, the specifications were mainly written by these members

Service operation and plan during project - We decided not
to make new changes for old system - A planner also participated in daily meeting - => We can change specs immediately J

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!
2021-08 Complete other apps development 2021-01 Start project Complete core app development

Difficulties and Benefits in the project 2021-09 Integration and QA
2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project

2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project

2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Reflections that came to light

2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light

2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light Benefits of rewriting

2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Reflections that came to light Benefits of rewriting Difficulties that occur early in the project

#1: Consider features to throw away - Features that are
no longer in use - Features that are still in use but implemented in a non-ideal way due to man-hours during implementation Difficulties that occur early in the project

#2: Sharing about discarded features Difficulties that occur early in
the project - What happened? - Discarded features were not shared well and were sometimes implemented - Why did it happen? - The features to be implemented were written in the specification, but the features to be discarded were not - What we do - Write about features to be discarded - Add comments to Perl code to remove it

Difficulties that occur early in the project - #1: Consider
features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers

#3: Difficulties of Perl for Kotlin engineers - Why do
Kotlin engineers need to read Perl? - Reason: the specs aren't completely documented L - Why the specs aren’t documented? - Reason1: the speed of adding/changing features was fast - we didn’t have time and human resources to organize specs on the wiki - Reason2: special features made for spot needs Difficulties that occur early in the project

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators
- $ @ % - Hard to see the contents of hashref

- $ @ % - Hard to see the contents of hashref Comment?

- $ @ % - Hard to see the contents of hashref It is defined-or!

- $ @ % - Hard to see the contents of hashref many $, @, %

- $ @ % - Hard to see the contents of hashref What is the content?

- $ @ % - Hard to see the contents of hashref You need to read many code L

Difficulties that occur early in the project - #1: Consider
features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers

#4: Difficulties of Spring Boot with Kotlin for Perl engineers
- Rather than Kotlin, the framework Spring Boot was difficult - I think Perl engineers became familiar with Kotlin language rather quickly J - NoClassDefFoundError - In Perl, you can import anything by just writing “import xxx;” - In Spring Boot, you need to learn Bean, DI, @ComponentScan…

Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light

Reflections (1/3): Too much use of nullable

Background: about variables in Perl - Variables in Perl are
similar to Kotlin nullable - Variables in Perl can be “undef” anywhere, so the defined check was always necessary - Therefore, when we rewrite Perl code for Kotlin, it tends to be written as nullable

About Kotlin nullable and non-null - non-null https://kotlinlang.org/docs/null-safety.html#checking-for-null-in-conditions - nullable

What is the problem with using nullable a lot? -
Problem1: More nullables means more null checks L - Problem2: We need to consider how to convert nullable to non-null - !! operator need to be used with care // throw NullPointerException If b is null https://kotlinlang.org/docs/null-safety.html#the-operator

Not good example of using !! operator A null check
written away from the !! operator

Not good example of using !! operator Let’s think about
adding another method like this NullPointerException may occur!

What should we do? - What should we do? -
We should use non-null as possible - !! operator need to be used with care - But, nullable made it easy to rewrite from Perl to Kotlin J

Reflection (2/3) : Handling of similar variables

We have variables “digest” with multiple meanings - What is
“digest”? - Tokens issued for each combination of user and campaign - There are “digest” with multiple meanings - Limited digest - Unlimited digest - LINE digest - External LINE digest - In the beginning, we handled all of the above digests with val digest: String - As a result, we couldn't figure out which digest was in what format L

What we actually do - We actually start to use
typealias - Just using typealias has improved readability

What should we do? We should have used inline class
https://kotlinlang.org/docs/inline-classes.html

Reflection (3/3) : Business logic knows outer world

What was the situation? Business logic knows outer world -
Some business logics throwed StatusRuntimeException which is in io.grpc - In other words, business logics knew how to communicate with the outside

A problem that we encountered Business logic knows outer world
- We wanted to use the logic from another application that didn’t use gRPC - Should we have added io.grpc as a dependency of the another app?? - No - Instead, we should have avoided using io.grpc.StatusRuntimeException in the common logic - As a result, it was hard to make those logics common L

Why were we in the situation? Business logic knows outer
world - In the early stages of the project, we were writing codes focused on the normal behavior of the system - We didn't pay enough attention to the exception behavior - We weren't aware of removing gRPC dependent code from business logic - Not enough thought was given to making logics common to applications without gRPC - And in the end, when we wanted to make the logics common, we didn't have time to clean it up

What should we do? Business logic knows outer world -
In our case - We should avoid using io.grpc.StatusRuntimeException in the common logic - We should separate the layers that know how to communicate with the outside world and those that don’t - To generalize, business logic should not know outer world

Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release

#1: REAL QA considering the Perl system - REAL QA
is easy if we replaced all the DBs - but that wasn't the case here - We decided that it would be best to use DBs in the Perl era - We had to carefully consider the release procedure, taking into account the impact on the current system - We’ve been running Perl and Kotlin in BETA for a long time to see if it generates incompatible data - We’ve verified that the number of MySQL connections from the new server is acceptable Difficulties related to release

#2: Develop the parts left in Perl - There are
parts of Perl that were left out of the initial release - It would be nice if we could replace everything in one release but that's not going to happen this time - If we replace everything in on release, we needed another 3 or 4 months L - But it's tough to stop other requests for services until then Difficulties related to release

Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release Benefits of rewriting

Benefits of rewriting - Unused features can be refreshed J
- Specs becomes not to depend on a particular employee J - Kotlin is commonly used in our development organization - Exchange knowledge J - Allocation of engineering resources J - Update the skill sets of engineers J

Conclusion - We rewrite 8-year-old Perl product with Kotlin -
The longer the service continues, the more complex the system becomes - I introduced the difficulty of rewriting a language and how to solve it - There is many insights and benefits gained from the project to rewrite the language J

Story of Rewriting 8-Year-Old Perl Product with...

Story of Rewriting 8-Year-Old Perl Product with Kotlin

Transcript

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Featured