Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Story of Rewriting 8-Year-Old Perl Product with Kotlin

Story of Rewriting 8-Year-Old Perl Product with Kotlin

LINE DEVDAY 2021

November 11, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. About Me Kohei Ohara - Joined LINE in 2018 -

    Server-side engineer of LINE Point Club - Writing code in Perl, Kotlin, and Java
  2. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  3. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  4. How much traffic Req / Sec - Avg 700 Req

    / Day 57M Req / Sec - Peak 15K
  5. System overview in Perl era cv-worker cv-worker apps for Use

    Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer
  6. System overview in Perl era cv-worker cv-worker apps for Use

    Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06
  7. System overview in Perl era cv-worker cv-worker apps for Use

    Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06 Launched in 2015-04
  8. System overview in Perl era cv-worker cv-worker apps for Use

    Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Difficulties of having 2 different language apps • Allocation of engineering resources L • Develop the same features in different languages L
  9. System overview in Perl era cv-worker cv-worker apps for Use

    Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer I’ll explain more
  10. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
  11. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Amon2 (Perl)
  12. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Spring Boot (Kotlin/Java)
  13. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway We cannot use HBase from Perl directly L
  14. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
  15. System overview in Perl era (Earn Tab) media admin cv-worker

    Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway
  16. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  17. Why Kotlin? - Most Point Club apps other than Perl

    are written in Kotlin Why not Java? Java/Kotlin is most used language in the company - Engineer resource J - There is a Java/Kotlin ecosystem in the company J
  18. Why did we decide to rewrite Perl to Kotlin? -

    Complicated system architecture L - Earn tab is implemented in Perl, use tab is implemented in Kotlin - HBase gateway app for Perl - Complicated features L - Many features that are no longer in use - Due to the small number of Perl engineers, it seemed impossible to continue development in the future L - Before the start of the project, there seemed to be no major changes for the next six months J
  19. Why did we decide to rewrite Perl to Kotlin? -

    The number of people maintaining Perl libraries is decreasing - Perl‘s client libraries for recent middleware are not rich - HBase - We have tried to use HBase from Perl several times - But we couldn't get it to work as we'd hoped - We ended up implementing HBase gateway in Java/Kotlin - Redis Cluster
  20. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  21. System overview after remaking media core admin cv-worker Kafka Redis

    Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  22. System overview after remaking media core admin cv-worker Kafka Redis

    Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Spring Boot (Kotlin)
  23. System overview after remaking media core admin cv-worker Kafka Redis

    Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Same DB as old system
  24. System overview: gRPC media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  25. System overview: Kafka as job queue media core admin cv-worker

    Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  26. System overview: HBase media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  27. System overview: HBase media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL HBase gateway is no longer needed J
  28. System overview: admin-app media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  29. System overview: admin-app media core cv-worker Kafka Redis Clusters Redis

    gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin
  30. System overview: admin-app media core cv-worker Kafka Redis Clusters Redis

    gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin
  31. System overview: media-app media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  32. System overview: media-app media core admin cv-worker Kafka Redis Clusters

    Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL media-app can use the data from both the earn and use tabs J
  33. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  34. Remake project developers 3 Java/Kotlin engineers - They have a

    lot of experience in developing for Java/Kotlin and Spring Boot - They don’t know much about the specification of earn tab in the Perl era - In the early days of the project, they were working on the base of the development - Without them, we could not have completed the Remake Project J 3 Perl engineers - They know about the specifications of the earn tab - In the early days of the project, the specifications were mainly written by these members
  35. Service operation and plan during project - We decided not

    to make new changes for old system - A planner also participated in daily meeting - => We can change specs immediately J
  36. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  37. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  38. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  39. System overview after remaking media core admin cv-worker Kafka Redis

    Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  40. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  41. System overview after remaking media core admin cv-worker Kafka Redis

    Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL
  42. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  43. Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release!

    2021-08 Complete other apps development 2021-01 Start project Complete core app development
  44. Agenda - About LINE Point Club - Why did we

    decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project
  45. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project
  46. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project
  47. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Reflections that came to light
  48. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light
  49. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light Benefits of rewriting
  50. Difficulties and Benefits in the project 2021-09 Integration and QA

    2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Reflections that came to light Benefits of rewriting Difficulties that occur early in the project
  51. #1: Consider features to throw away - Features that are

    no longer in use - Features that are still in use but implemented in a non-ideal way due to man-hours during implementation Difficulties that occur early in the project
  52. #2: Sharing about discarded features Difficulties that occur early in

    the project - What happened? - Discarded features were not shared well and were sometimes implemented - Why did it happen? - The features to be implemented were written in the specification, but the features to be discarded were not - What we do - Write about features to be discarded - Add comments to Perl code to remove it
  53. Difficulties that occur early in the project - #1: Consider

    features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers
  54. #3: Difficulties of Perl for Kotlin engineers - Why do

    Kotlin engineers need to read Perl? - Reason: the specs aren't completely documented L - Why the specs aren’t documented? - Reason1: the speed of adding/changing features was fast - we didn’t have time and human resources to organize specs on the wiki - Reason2: special features made for spot needs Difficulties that occur early in the project
  55. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref
  56. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref Comment?
  57. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref It is defined-or!
  58. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref many $, @, %
  59. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref What is the content?
  60. #3: Difficulties of Perl for Kotlin engineers - Defined-or operators

    - $ @ % - Hard to see the contents of hashref You need to read many code L
  61. Difficulties that occur early in the project - #1: Consider

    features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers
  62. #4: Difficulties of Spring Boot with Kotlin for Perl engineers

    - Rather than Kotlin, the framework Spring Boot was difficult - I think Perl engineers became familiar with Kotlin language rather quickly J - NoClassDefFoundError - In Perl, you can import anything by just writing “import xxx;” - In Spring Boot, you need to learn Bean, DI, @ComponentScan…
  63. Difficulties and Benefits in the project 2021-09 Integration and QA

    Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light
  64. Background: about variables in Perl - Variables in Perl are

    similar to Kotlin nullable - Variables in Perl can be “undef” anywhere, so the defined check was always necessary - Therefore, when we rewrite Perl code for Kotlin, it tends to be written as nullable
  65. What is the problem with using nullable a lot? -

    Problem1: More nullables means more null checks L - Problem2: We need to consider how to convert nullable to non-null - !! operator need to be used with care // throw NullPointerException If b is null https://kotlinlang.org/docs/null-safety.html#the-operator
  66. Not good example of using !! operator A null check

    written away from the !! operator
  67. Not good example of using !! operator Let’s think about

    adding another method like this NullPointerException may occur!
  68. What should we do? - What should we do? -

    We should use non-null as possible - !! operator need to be used with care - But, nullable made it easy to rewrite from Perl to Kotlin J
  69. We have variables “digest” with multiple meanings - What is

    “digest”? - Tokens issued for each combination of user and campaign - There are “digest” with multiple meanings - Limited digest - Unlimited digest - LINE digest - External LINE digest - In the beginning, we handled all of the above digests with val digest: String - As a result, we couldn't figure out which digest was in what format L
  70. What we actually do - We actually start to use

    typealias - Just using typealias has improved readability
  71. What should we do? We should have used inline class

    https://kotlinlang.org/docs/inline-classes.html
  72. What was the situation? Business logic knows outer world -

    Some business logics throwed StatusRuntimeException which is in io.grpc - In other words, business logics knew how to communicate with the outside
  73. A problem that we encountered Business logic knows outer world

    - We wanted to use the logic from another application that didn’t use gRPC - Should we have added io.grpc as a dependency of the another app?? - No - Instead, we should have avoided using io.grpc.StatusRuntimeException in the common logic - As a result, it was hard to make those logics common L
  74. Why were we in the situation? Business logic knows outer

    world - In the early stages of the project, we were writing codes focused on the normal behavior of the system - We didn't pay enough attention to the exception behavior - We weren't aware of removing gRPC dependent code from business logic - Not enough thought was given to making logics common to applications without gRPC - And in the end, when we wanted to make the logics common, we didn't have time to clean it up
  75. What should we do? Business logic knows outer world -

    In our case - We should avoid using io.grpc.StatusRuntimeException in the common logic - We should separate the layers that know how to communicate with the outside world and those that don’t - To generalize, business logic should not know outer world
  76. Difficulties and Benefits in the project 2021-09 Integration and QA

    Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release
  77. #1: REAL QA considering the Perl system - REAL QA

    is easy if we replaced all the DBs - but that wasn't the case here - We decided that it would be best to use DBs in the Perl era - We had to carefully consider the release procedure, taking into account the impact on the current system - We’ve been running Perl and Kotlin in BETA for a long time to see if it generates incompatible data - We’ve verified that the number of MySQL connections from the new server is acceptable Difficulties related to release
  78. #2: Develop the parts left in Perl - There are

    parts of Perl that were left out of the initial release - It would be nice if we could replace everything in one release but that's not going to happen this time - If we replace everything in on release, we needed another 3 or 4 months L - But it's tough to stop other requests for services until then Difficulties related to release
  79. Difficulties and Benefits in the project 2021-09 Integration and QA

    Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release Benefits of rewriting
  80. Benefits of rewriting - Unused features can be refreshed J

    - Specs becomes not to depend on a particular employee J - Kotlin is commonly used in our development organization - Exchange knowledge J - Allocation of engineering resources J - Update the skill sets of engineers J
  81. Conclusion - We rewrite 8-year-old Perl product with Kotlin -

    The longer the service continues, the more complex the system becomes - I introduced the difficulty of rewriting a language and how to solve it - There is many insights and benefits gained from the project to rewrite the language J