Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

About Me Kohei Ohara - Joined LINE in 2018 - Server-side engineer of LINE Point Club - Writing code in Perl, Kotlin, and Java

Slide 3

Slide 3 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 4

Slide 4 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 5

Slide 5 text

LINE Point Club

Slide 6

Slide 6 text

- You can earn ”LINE Point” Earn Tab

Slide 7

Slide 7 text

- You can exchange ”LINE Point” Use Tab

Slide 8

Slide 8 text

How much traffic Req / Sec - Avg 700 Req / Day 57M Req / Sec - Peak 15K

Slide 9

Slide 9 text

System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer

Slide 10

Slide 10 text

System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06

Slide 11

Slide 11 text

System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Launched in 2013-06 Launched in 2015-04

Slide 12

Slide 12 text

System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer Difficulties of having 2 different language apps • Allocation of engineering resources L • Develop the same features in different languages L

Slide 13

Slide 13 text

System overview in Perl era cv-worker cv-worker apps for Use Tab DB for Use Tab Use Tab (Kotlin) cv-worker cv-worker apps for Earn Tab DB for Earn Tab Earn Tab (Perl) Reverse Proxy nginx load balancer I’ll explain more

Slide 14

Slide 14 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway

Slide 15

Slide 15 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Amon2 (Perl)

Slide 16

Slide 16 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway Spring Boot (Kotlin/Java)

Slide 17

Slide 17 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway We cannot use HBase from Perl directly L

Slide 18

Slide 18 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway

Slide 19

Slide 19 text

System overview in Perl era (Earn Tab) media admin cv-worker Redis MySQL HBase REST MySQL for job queue Memcached REST REST REST HBase gateway

Slide 20

Slide 20 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 21

Slide 21 text

Why Kotlin? - Most Point Club apps other than Perl are written in Kotlin Why not Java? Java/Kotlin is most used language in the company - Engineer resource J - There is a Java/Kotlin ecosystem in the company J

Slide 22

Slide 22 text

Why did we decide to rewrite Perl to Kotlin? - Complicated system architecture L - Earn tab is implemented in Perl, use tab is implemented in Kotlin - HBase gateway app for Perl - Complicated features L - Many features that are no longer in use - Due to the small number of Perl engineers, it seemed impossible to continue development in the future L - Before the start of the project, there seemed to be no major changes for the next six months J

Slide 23

Slide 23 text

Why did we decide to rewrite Perl to Kotlin? - The number of people maintaining Perl libraries is decreasing - Perl‘s client libraries for recent middleware are not rich - HBase - We have tried to use HBase from Perl several times - But we couldn't get it to work as we'd hoped - We ended up implementing HBase gateway in Java/Kotlin - Redis Cluster

Slide 24

Slide 24 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 25

Slide 25 text

System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 26

Slide 26 text

System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Spring Boot (Kotlin)

Slide 27

Slide 27 text

System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL Same DB as old system

Slide 28

Slide 28 text

System overview: gRPC media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 29

Slide 29 text

System overview: Kafka as job queue media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 30

Slide 30 text

System overview: HBase media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 31

Slide 31 text

System overview: HBase media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL HBase gateway is no longer needed J

Slide 32

Slide 32 text

System overview: admin-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 33

Slide 33 text

System overview: admin-app media core cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin

Slide 34

Slide 34 text

System overview: admin-app media core cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL admin

Slide 35

Slide 35 text

System overview: media-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 36

Slide 36 text

System overview: media-app media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL media-app can use the data from both the earn and use tabs J

Slide 37

Slide 37 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 38

Slide 38 text

Remake project developers 3 Java/Kotlin engineers - They have a lot of experience in developing for Java/Kotlin and Spring Boot - They don’t know much about the specification of earn tab in the Perl era - In the early days of the project, they were working on the base of the development - Without them, we could not have completed the Remake Project J 3 Perl engineers - They know about the specifications of the earn tab - In the early days of the project, the specifications were mainly written by these members

Slide 39

Slide 39 text

Service operation and plan during project - We decided not to make new changes for old system - A planner also participated in daily meeting - => We can change specs immediately J

Slide 40

Slide 40 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 41

Slide 41 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 42

Slide 42 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 43

Slide 43 text

System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 44

Slide 44 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 45

Slide 45 text

System overview after remaking media core admin cv-worker Kafka Redis Clusters Redis gRPC gRPC media DB DB for Earn Tab MySQL MySQL HBase REST DB for Use Tab MySQL

Slide 46

Slide 46 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 47

Slide 47 text

Remake project schedule 2021-09 Integration and QA 2021-04 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Complete core app development

Slide 48

Slide 48 text

Agenda - About LINE Point Club - Why did we decide to rewrite Perl to Kotlin? - System overview after remaking - How to proceed with the project - Difficulties and Benefits in the project

Slide 49

Slide 49 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project

Slide 50

Slide 50 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project

Slide 51

Slide 51 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Reflections that came to light

Slide 52

Slide 52 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light

Slide 53

Slide 53 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties that occur early in the project Difficulties related to release Reflections that came to light Benefits of rewriting

Slide 54

Slide 54 text

Difficulties and Benefits in the project 2021-09 Integration and QA 2021-04 Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Reflections that came to light Benefits of rewriting Difficulties that occur early in the project

Slide 55

Slide 55 text

#1: Consider features to throw away - Features that are no longer in use - Features that are still in use but implemented in a non-ideal way due to man-hours during implementation Difficulties that occur early in the project

Slide 56

Slide 56 text

#2: Sharing about discarded features Difficulties that occur early in the project - What happened? - Discarded features were not shared well and were sometimes implemented - Why did it happen? - The features to be implemented were written in the specification, but the features to be discarded were not - What we do - Write about features to be discarded - Add comments to Perl code to remove it

Slide 57

Slide 57 text

Difficulties that occur early in the project - #1: Consider features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers

Slide 58

Slide 58 text

#3: Difficulties of Perl for Kotlin engineers - Why do Kotlin engineers need to read Perl? - Reason: the specs aren't completely documented L - Why the specs aren’t documented? - Reason1: the speed of adding/changing features was fast - we didn’t have time and human resources to organize specs on the wiki - Reason2: special features made for spot needs Difficulties that occur early in the project

Slide 59

Slide 59 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref

Slide 60

Slide 60 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref Comment?

Slide 61

Slide 61 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref It is defined-or!

Slide 62

Slide 62 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref many $, @, %

Slide 63

Slide 63 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref What is the content?

Slide 64

Slide 64 text

#3: Difficulties of Perl for Kotlin engineers - Defined-or operators - $ @ % - Hard to see the contents of hashref You need to read many code L

Slide 65

Slide 65 text

Difficulties that occur early in the project - #1: Consider features to throw away - #2: Sharing about discarded features - #3: Difficulties of Perl for Kotlin engineers - #4: Difficulties of Spring Boot with Kotlin for Perl engineers

Slide 66

Slide 66 text

#4: Difficulties of Spring Boot with Kotlin for Perl engineers - Rather than Kotlin, the framework Spring Boot was difficult - I think Perl engineers became familiar with Kotlin language rather quickly J - NoClassDefFoundError - In Perl, you can import anything by just writing “import xxx;” - In Spring Boot, you need to learn Bean, DI, @ComponentScan…

Slide 67

Slide 67 text

Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Difficulties related to release Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light

Slide 68

Slide 68 text

Reflections (1/3): Too much use of nullable

Slide 69

Slide 69 text

Background: about variables in Perl - Variables in Perl are similar to Kotlin nullable - Variables in Perl can be “undef” anywhere, so the defined check was always necessary - Therefore, when we rewrite Perl code for Kotlin, it tends to be written as nullable

Slide 70

Slide 70 text

About Kotlin nullable and non-null - non-null https://kotlinlang.org/docs/null-safety.html#checking-for-null-in-conditions - nullable

Slide 71

Slide 71 text

What is the problem with using nullable a lot? - Problem1: More nullables means more null checks L - Problem2: We need to consider how to convert nullable to non-null - !! operator need to be used with care // throw NullPointerException If b is null https://kotlinlang.org/docs/null-safety.html#the-operator

Slide 72

Slide 72 text

Not good example of using !! operator A null check written away from the !! operator

Slide 73

Slide 73 text

Not good example of using !! operator Let’s think about adding another method like this NullPointerException may occur!

Slide 74

Slide 74 text

What should we do? - What should we do? - We should use non-null as possible - !! operator need to be used with care - But, nullable made it easy to rewrite from Perl to Kotlin J

Slide 75

Slide 75 text

Reflection (2/3) : Handling of similar variables

Slide 76

Slide 76 text

We have variables “digest” with multiple meanings - What is “digest”? - Tokens issued for each combination of user and campaign - There are “digest” with multiple meanings - Limited digest - Unlimited digest - LINE digest - External LINE digest - In the beginning, we handled all of the above digests with val digest: String - As a result, we couldn't figure out which digest was in what format L

Slide 77

Slide 77 text

What we actually do - We actually start to use typealias - Just using typealias has improved readability

Slide 78

Slide 78 text

What should we do? We should have used inline class https://kotlinlang.org/docs/inline-classes.html

Slide 79

Slide 79 text

Reflection (3/3) : Business logic knows outer world

Slide 80

Slide 80 text

What was the situation? Business logic knows outer world - Some business logics throwed StatusRuntimeException which is in io.grpc - In other words, business logics knew how to communicate with the outside

Slide 81

Slide 81 text

A problem that we encountered Business logic knows outer world - We wanted to use the logic from another application that didn’t use gRPC - Should we have added io.grpc as a dependency of the another app?? - No - Instead, we should have avoided using io.grpc.StatusRuntimeException in the common logic - As a result, it was hard to make those logics common L

Slide 82

Slide 82 text

Why were we in the situation? Business logic knows outer world - In the early stages of the project, we were writing codes focused on the normal behavior of the system - We didn't pay enough attention to the exception behavior - We weren't aware of removing gRPC dependent code from business logic - Not enough thought was given to making logics common to applications without gRPC - And in the end, when we wanted to make the logics common, we didn't have time to clean it up

Slide 83

Slide 83 text

What should we do? Business logic knows outer world - In our case - We should avoid using io.grpc.StatusRuntimeException in the common logic - We should separate the layers that know how to communicate with the outside world and those that don’t - To generalize, business logic should not know outer world

Slide 84

Slide 84 text

Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project Benefits of rewriting 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release

Slide 85

Slide 85 text

#1: REAL QA considering the Perl system - REAL QA is easy if we replaced all the DBs - but that wasn't the case here - We decided that it would be best to use DBs in the Perl era - We had to carefully consider the release procedure, taking into account the impact on the current system - We’ve been running Perl and Kotlin in BETA for a long time to see if it generates incompatible data - We’ve verified that the number of MySQL connections from the new server is acceptable Difficulties related to release

Slide 86

Slide 86 text

#2: Develop the parts left in Perl - There are parts of Perl that were left out of the initial release - It would be nice if we could replace everything in one release but that's not going to happen this time - If we replace everything in on release, we needed another 3 or 4 months L - But it's tough to stop other requests for services until then Difficulties related to release

Slide 87

Slide 87 text

Difficulties and Benefits in the project 2021-09 Integration and QA Complete core app development 2021-09-27 Release! 2021-08 Complete other apps development 2021-01 Start project 2021-04 Difficulties that occur early in the project Reflections that came to light Difficulties related to release Benefits of rewriting

Slide 88

Slide 88 text

Benefits of rewriting - Unused features can be refreshed J - Specs becomes not to depend on a particular employee J - Kotlin is commonly used in our development organization - Exchange knowledge J - Allocation of engineering resources J - Update the skill sets of engineers J

Slide 89

Slide 89 text

Conclusion - We rewrite 8-year-old Perl product with Kotlin - The longer the service continues, the more complex the system becomes - I introduced the difficulty of rewriting a language and how to solve it - There is many insights and benefits gained from the project to rewrite the language J