Story of Rewriting 8-Year-Old Perl Product with Kotlin

Kohei Ohara
LINE / OA Dev 3 Team / Software Engineer

https://linedevday.linecorp.com/2021/ja/sessions/157
https://linedevday.linecorp.com/2021/en/sessions/157
https://linedevday.linecorp.com/2021/ko/sessions/157

Transcript

2. About Me Kohei Ohara - Joined LINE in 2018 - Server-side engineer of LINE Point Club - Writing code in Perl, Kotlin, and Java
3. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
4. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
5. LINE Point Club
6. - You can earn ”LINE Point” Earn Tab
7. - You can exchange ”LINE Point” Use Tab
8. How much traffic Req / Sec - Avg 700 Req / Day 57M Req / Sec - Peak 15K
9. System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer
10. System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06
11. System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06 Launched in 2015-04
12. System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Difficulties of having 2 different language apps • Allocation of engineering resources L • Develop the same features in different languages L
13. System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer I’ll explain more
14. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
15. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Amon2 (Perl)
16. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Spring Boot (Kotlin/Java)
17. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway We cannot use HBase from Perl directly L
18. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
19. System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
20. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
21. Why Kotlin? - Most Point Club apps other than Perl are written in Kotlin Why not Java? Java/Kotlin is most used language in the company - Engineer resource J - There is a Java/Kotlin ecosystem in the company J
22. Why did we decide to rewrite Perl to Kotlin? - Complicated system architecture L - Earn tab is implemented in Perl, use tab is implemented in Kotlin - HBase gateway app for Perl - Complicated features L - Many features that are no longer in use - Due to the small number of Perl engineers, it seemed impossible to continue development in the future L - Before the start of the project, there seemed to be no major changes for the next six months J
23. Why did we decide to rewrite Perl to Kotlin? - The number of people maintaining Perl libraries is decreasing - Perl‘s client libraries for recent middleware are not rich - HBase - We have tried to use HBase from Perl several times - But we couldn't get it to work as we'd hoped - We ended up implementing HBase gateway in Java/Kotlin - Redis Cluster
24. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
25. System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
26. System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Spring Boot (Kotlin)
27. System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Same DB as old system
28. System overview: gRPC media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
29. System overview: Kafka as job queue media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
30. System overview: HBase media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
31. System overview: HBase media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL HBase gateway is no longer needed J
32. System overview: admin-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
33. System overview: admin-app media core cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin
34. System overview: admin-app media core cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin
35. System overview: media-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
36. System overview: media-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL media-app can use the data from both the earn and use tabs J
37. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
38. Remake project developers 3 Java/Kotlin engineers - They have a lot of experience in developing for Java/Kotlin and Spring Boot - They don’t know much about the specification of earn tab in the Perl era - In the early days of the project, they were working on the base of the development - Without them, we could not have completed the Remake Project J 3 Perl engineers - They know about the specifications of the earn tab - In the early days of the project, the specifications were mainly written by these members
39. Service operation and plan during project - We decided not to make new changes for old system - A planner also participated in daily meeting - => We can change specs immediately J
40. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
41. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
42. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
43. System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
44. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
45. System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
46. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
47. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development
48. Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
49. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project
50. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project
51. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Reflections that came to light
52. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light
53. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light Benefits of rewriting
54. Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Reflections that came to light Benefits of rewriting Difficulties that occur early in the project
55. #1: Consider features to throw away - Features that are no longer in use - Features that are still in use but implemented in a non-ideal way due to man-hours during implementation Difficulties that occur early in the project
56. #2: Sharing about discarded features Difficulties that occur early in the project - What happened? - Discarded features were not shared well and were sometimes implemented - Why did it happen? - The features to be implemented were written in the specification, but the features to be discarded were not - What we do - Write about features to be discarded - Add comments to Perl code to remove it
57. Difficulties that occur early in the project - #1: Consider features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers
58. #3: Difficulties of Perl for Kotlin engineers - Why do Kotlin engineers need to read Perl? - Reason: the specs aren't completely documented L - Why the specs aren’t documented? - Reason1: the speed of adding/changing features was fast - we didn’t have time and human resources to organize specs on the wiki - Reason2: special features made for spot needs Difficulties that occur early in the project
59. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref
60. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref Comment?
61. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref It is defined-or!
62. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref many $, @, %
63. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref What is the content?
64. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref You need to read many code L
65. Difficulties that occur early in the project - #1: Consider features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers
66. #4: Difficulties of Spring Boot with Kotlin for Perl engineers - Rather than Kotlin, the framework Spring Boot was difficult - I think Perl engineers became familiar with Kotlin language rather quickly J - NoClassDefFoundError - In Perl, you can import anything by just writing “import xxx;” - In Spring Boot, you need to learn Bean, DI, @ComponentScan…
67. Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light
68. Reflections (1/3): Too much use of nullable
69. Background: about variables in Perl - Variables in Perl are similar to Kotlin nullable - Variables in Perl can be “undef” anywhere, so the defined check was always necessary - Therefore, when we rewrite Perl code for Kotlin, it tends to be written as nullable
70. About Kotlin nullable and non-null - non-null https://kotlinlang.org/docs/null-safety.html#checking-for-null-in-conditions - nullable
71. What is the problem with using nullable a lot? - Problem1: More nullables means more null checks L - Problem2: We need to consider how to convert nullable to non-null - !! operator need to be used with care // throw NullPointerException If b is null https://kotlinlang.org/docs/null-safety.html#the-operator
72. Not good example of using !! operator A null check written away from the !! operator
73. Not good example of using !! operator Let’s think about adding another method like this NullPointerException may occur!
74. What should we do? - What should we do? - We should use non-null as possible - !! operator need to be used with care - But, nullable made it easy to rewrite from Perl to Kotlin J
75. Reflection (2/3) : Handling of similar variables
76. We have variables “digest” with multiple meanings - What is “digest”? - Tokens issued for each combination of user and campaign - There are “digest” with multiple meanings - Limited digest - Unlimited digest - LINE digest - External LINE digest - In the beginning, we handled all of the above digests with val digest: String - As a result, we couldn't figure out which digest was in what format L
77. What we actually do - We actually start to use typealias - Just using typealias has improved readability
78. What should we do? We should have used inline class https://kotlinlang.org/docs/inline-classes.html
79. Reflection (3/3) : Business logic knows outer world
80. What was the situation? Business logic knows outer world - Some business logics throwed StatusRuntimeException which is in io.grpc - In other words, business logics knew how to communicate with the outside
81. A problem that we encountered Business logic knows outer world - We wanted to use the logic from another application that didn’t use gRPC - Should we have added io.grpc as a dependency of the another app?? - No - Instead, we should have avoided using io.grpc.StatusRuntimeException in the common logic - As a result, it was hard to make those logics common L
82. Why were we in the situation? Business logic knows outer world - In the early stages of the project, we were writing codes focused on the normal behavior of the system - We didn't pay enough attention to the exception behavior - We weren't aware of removing gRPC dependent code from business logic - Not enough thought was given to making logics common to applications without gRPC - And in the end, when we wanted to make the logics common, we didn't have time to clean it up
83. What should we do? Business logic knows outer world - In our case - We should avoid using io.grpc.StatusRuntimeException in the common logic - We should separate the layers that know how to communicate with the outside world and those that don’t - To generalize, business logic should not know outer world
84. Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release
85. #1: REAL QA considering the Perl system - REAL QA is easy if we replaced all the DBs - but that wasn't the case here - We decided that it would be best to use DBs in the Perl era - We had to carefully consider the release procedure, taking into account the impact on the current system - We’ve been running Perl and Kotlin in BETA for a long time to see if it generates incompatible data - We’ve verified that the number of MySQL connections from the new server is acceptable Difficulties related to release
86. #2: Develop the parts left in Perl - There are parts of Perl that were left out of the initial release - It would be nice if we could replace everything in one release but that's not going to happen this time - If we replace everything in on release, we needed another 3 or 4 months L - But it's tough to stop other requests for services until then Difficulties related to release
87. Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release Benefits of rewriting
88. Benefits of rewriting - Unused features can be refreshed J - Specs becomes not to depend on a particular employee J - Kotlin is commonly used in our development organization - Exchange knowledge J - Allocation of engineering resources J - Update the skill sets of engineers J
89. Conclusion - We rewrite 8-year-old Perl product with Kotlin - The longer the service continues, the more complex the system becomes - I introduced the difficulty of rewriting a language and how to solve it - There is many insights and benefits gained from the project to rewrite the language J