Using Akka Cluster for a payment service

by Kazuki Negoro

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

I ll talk about... Technical story of TIS Participates in Demonstration Trial of QR Payment Using High Availablity and High Throughput Software Stack Lerna - Applied to Omotenashi Platform for foreigh tourists visiting Japan - © 2019 TIS Inc. TIS、高可用・高スループットのソフトウェアスタック「Lerna」を活用したQR決済の実証実験に参画～訪日外国人観光客向け「おもてなしプラットフォーム」に適用～今日話すこと 2019/3/27 ニュースリリースの技術的な裏側 3 / 73

Slide 4

Slide 4 text

Omotenashi Platform A measure promoted by Ministry of Economy to increase the satisfication of foreign tourists. QR code payment service sightseeing information service etc... © 2019 TIS Inc. withJorge - Japan Omotenashi concieRGE 経済産業省が推進する、訪日外国人の満足度を高めるための施策 QRコード決済や観光情報提供などのサービスが受けられる 4 / 73

Slide 5

Slide 5 text

Akka Apache Cassandra MariaDB Keepalived HAProxy What is Lerna ? A Software Stack for High Availablity and High Throughput (Reactive System) which is promoted by TIS © 2019 TIS Inc. Lerna とは何か？ TIS が推進している高可用・高スループットを実現するためのソフトウェアスタック 5 / 73

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Main features of the system QR Payment + Payment history confirmation Cancelling payment Reconstructed these features with Akka Cluster Sharding and EC + CQRS © 2019 TIS Inc. システムの主要機能はQRコード決済、決済履歴確認、決済キャンセル。これら主要機能を Lerna の構成にリプレース。 11 / 73

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Why is this architecture needed ? RDBMS makes it difficult to scale writes Using ES + CQRS makes writing easier to scale © 2019 TIS Inc. RDBMSは書き込みがボトルネックになりやすい ES + CQRS を使って書込・読込の両方をスケールできるように 19 / 73

Slide 20

Slide 20 text

Event Sourcing ? Persist events produced inside a system Only INSERT events to datastore Account No:10Opened Event System State ￥2,000Withdrew No:10, Balance:￥18,000 ￥10,000Deposited ￥10,000Deposited © 2019 TIS Inc. システムの中で起きたイベントを永続化イベントをデータストアに INSERT だけする 20 / 73

Slide 21

Slide 21 text

Pros of Event Sourcing Events is immutable No need lock・Can write data to multi nodes Easy to scale for to write data Easy to caching, copying, sharing Easy to scala for to read data, too © 2019 TIS Inc. イベントは不変なのでロック不要で複数のノードに分散書き込みが可能。複製も容易なので読み込みもスケール可能。 21 / 73

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Pros of CQRS Command Query Command Query different Database and models Command-side uses Event Sourcing. Persist on the Query-side in a form easy to sum and list © 2019 TIS Inc. コマンドとクエリで異なるDB・データ構造が使える。クエリサイドには集計や一覧しやすい形で永続化する。 25 / 73

Slide 26

Slide 26 text

Cons of CQRS Query-Side is updated asynchronus There is NO strong consistency between Command-Side and Query-Side However, sometime it will be updated: eventual consistency © 2019 TIS Inc. クエリサイドへの反映は非同期のためコマンドサイドの更新は遅れてクエリサイドに反映される（結果整合性） 26 / 73

Slide 27

Slide 27 text

Cons of CQRS Query-Side is updated asynchronus There is NO strong consistency between Command-Side and Query-Side However, sometime it will be updated: eventual consistency Need to design the system or business so that we dont have to worry about temporary inconsistencies © 2019 TIS Inc. クエリサイドへの反映は非同期のためコマンドサイドの更新は遅れてクエリサイドに反映される（結果整合性）一時的に一貫性が取れていなくても問題ないようにビジネスやシステムを設計する必要がある 27 / 73

Slide 28

Slide 28 text

Not always acceptable eventual consistency Can not withdraw if the balance is not enough Only one person can reserve a seat One-time password can only be used once ... © 2019 TIS Inc. あらゆる場面で結果整合性を許容できるわけではない 28 / 73

Slide 29

Slide 29 text

Not always acceptable eventual consistency Can not withdraw if the balance is not enough Only one person can reserve a seat One-time password can only be used once ... Need to look the latest state without delay to realize these requirements © 2019 TIS Inc. あらゆる場面で結果整合性を許容できるわけではないこれらの要求を実現するには、遅延無く最新の状態を確認できる必要がある 29 / 73

Slide 30

Slide 30 text

How to look latest state Akka Persistence makes Actor to be able to persist events and look latest state (as self state) Account No:10Opened Event ActorState ￥2,000Withdrew No:10, Balance:￥18,000 ￥10,000Deposited ￥10,000Deposited © 2019 TIS Inc. Akka Persistence によって Actor はイベントの永続化と（それ自身の状態として）最新の状態の確認が可能になる 30 / 73

Slide 31

Slide 31 text

Slide 32

Slide 32 text

ES + CQRS Overview of Akka Command-Side Query-Side Event Store ReadModelUpdater Polling UpdateState Command Query APICall APICall Entity Entity Query Store Entity Controller © 2019 TIS Inc. Akka を用いた ES + CQRS の概要 32 / 73

Slide 33

Slide 33 text

Important point of Entity An Entity which has the ID must exists only one place Having one entity in multiple location cases inconsistency ID:1 ID:1 amount:10 amount:11 © 2019 TIS Inc. ある ID の Entity が存在していいのは 1 箇所でだけ複数箇所に存在するとデータの整合性が崩れる 33 / 73

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Scale Command-Side safely Akka Cluster Sharding Feature for load balancing Can distribute entities across several servers Guarantee that there is at most one entity in the cluster No single point of failure © 2019 TIS Inc. コマンドサイドを安全にスケールさせるには負荷分散の仕組み。複数のサーバーに Entity を分散できる各 Entity は高々1つだけ存在することが保証される。SPOFなし 36 / 73

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Decide where apply ES + CQRS to There is not many enginner who can use Scala/Akka Need many cost to change data model and paradigm Which do features need high availability and high throughput ? © 2019 TIS Inc. Scala/Akka のエンジニアが多いわけではない。データモデルとパラダイムの変更にはコストがかかる。 38 / 73

Slide 39

Slide 39 text

Decide where apply ES + CQRS to There is not many enginner who can use Scala/Akka Need many cost to change data model and paradigm Which do features need high availability and high throughput ? Apply only to features directly related to payment © 2019 TIS Inc. Scala/Akka のエンジニアが多いわけではない。データモデルとパラダイムの変更にはコストがかかる。決済に直接関係する機能にだけ適用 39 / 73

Slide 40

Slide 40 text

Consumer Merchant QR-Server GenQRCode QRCode Payment resultofpayment QRCode ReadQRCode Notification Consumer Merchant QR-Server Paymenthistory Cancel Getpaymenthistory resultofcancellation Payment flow © 2019 TIS Inc. 40 / 73

Slide 41

Slide 41 text

Requirements Sometime QR code is reused. Must not give same QR code multiple transactions at the same time. Consumer confirms payment progress Consumer and Merchant confirm payment history Merchant can cancel a payment © 2019 TIS Inc. QRコードは再利用される。顧客は決済の進捗が確認できる。決済履歴が確認できる。店舗は決済をキャンセルできる。 41 / 73

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Where is status updated? (Command-side) Record QR code usage status and expiration time Record the progress of payment Record results when payment completed Record results when payment canceling completed © 2019 TIS Inc. 状態が更新される箇所はどこか 43 / 73

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Decide implimentation methods of Queries Payment progress notification Reply the state of TransactionActor Payment history list Aggregate multiple TransactionActor states © 2019 TIS Inc. クエリの実装方法を決定。通知は TransactionActor の状態を返し、履歴一覧は複数の TransactionActor の状態を集約 46 / 73

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Design entities as a state machine TransactionActor WaitingForReadToken PreparingToPayment WaitingForPaymentExecution Finished WaitingForPaymentCancellation [？？？] [？？？] [？？？] [？？？] [？？？] [？？？][？？？] © 2019 TIS Inc. ステートマシンとして Entity を設計 48 / 73

Slide 49

Slide 49 text

TokenActor To use started Expired TransactionActor Token read Payment started Payment failed Payment completed Cancellation started ... Find events to change state of an entity © 2019 TIS Inc. エンティティの状態を変更するイベントを分析 49 / 73

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Apply events to state machines TransactionActor WaitingForReadToken PreparingToPayment WaitingForPaymentExecution Finished WaitingForPaymentCancellation [Tokengenerated] [Tokenread] [Paymentstarted] [Paymentfailed] [Paymentcompleted] [Paymentfailed] [Cancellationstarted] [Cancellationcompleted] [Cancellationfailed] © 2019 TIS Inc. ステートマシンにイベントを適用 51 / 73

Slide 52

Slide 52 text

Slide 53

Slide 53 text

How to construct payment history list TransactionActor knows a payment status. However, to simply collect the state of the entity is inefficient. © 2019 TIS Inc. TransactionActor は決済のステータスを知っているが、entity の状態を単純に集めるのは非効率 53 / 73

Slide 54

Slide 54 text

How to construct payment history list Construct payment hisotry list from events in Command-side to Query-side using akka-persistent-query © 2019 TIS Inc. Persist on the Query-side in a form easy to sum and list ❝ Command-Side のイベントを元に Query-Side へ決済履歴一覧を作成 54 / 73

Slide 55

Slide 55 text

How to construct payment history list Need to consider about the delay of Query-side updates. We can accept the delay because payment history list will be visible by user action. © 2019 TIS Inc. There is NO strong consistency between Command-Side and Query-Side ❝ Query-side への反映には遅延があるため許容できるか要検討。決済履歴一覧はユーザー操作によって開かれるため許容できた。 55 / 73

Slide 56

Slide 56 text

Reduce the delay until updated Query-side The delay is about 10 seconds by default. How does the delay affect the business ? Can not cancel payment Some delay is acceptable, but it is better to update faster © 2019 TIS Inc. クエリサイドが更新されるまでの遅延を短縮する。キャンセルできるようにできるだけ早く反映されるほうが望ましい。 56 / 73

Slide 57

Slide 57 text

Use pubsub-notification We could reduce the delay by Turning on of akka-persistence-cassandra. However, this option disabled guarantee to order events from multiple entities by time. © 2019 TIS Inc. akka-persistence-cassandra/reference.conf pubsub-notification という設定を有効にして遅延を短縮できたが、Entity を跨るイベントの順序が保証されなくなる。 57 / 73

Slide 58

Slide 58 text

How to get scalability and availability of Command-side Distribute TransactionActor and TokenActor across several server with Akka Cluster Sharding © 2019 TIS Inc. Akka Cluster Sharding を使って TransactionActor と TokenActor を複数サーバーに分散配置 58 / 73

Slide 59

Slide 59 text

Prepare for Split Brain A phenomenon in which a cluster is divided into multiple parts due to a network failure etc. This cause to lose consistency of the data. © 2019 TIS Inc. ネットワーク障害などによりクラスターが分割され、データの一貫性が損なわれたりする現象 59 / 73

Slide 60

Slide 60 text

Split Brain in Akka Cluster Sharding If split brain occurs in Akka Cluster Sharding, multiple Entities with the same ID will be created. Consistency is lost because the state of entity is not synchronized. © 2019 TIS Inc. Akka Cluster Sharding で Split Brain が発生すると同じ ID の Entity が複数生まれてしまい、一貫性が維持できなくなる 60 / 73

Slide 61

Slide 61 text

Split Brain in Akka Cluster Sharding Akka Cluster Shardings auto failover is disabled by default, which prevents Split Brain. But it will lower the availability. © 2019 TIS Inc. Akka Cluster Sharding の自動フェイルオーバーは Split Brain を防ぐためデフォルトで無効になっている。 61 / 73

Slide 62

Slide 62 text

Split Brain Solutions for Akka Cluster Split Brain Resolver (Lightbend: Akka Commercial Addons) TanUkkii007/akka-cluster-custom-downing mbilski/akka-reasonable-downing Stop one or all of the split clusters due to network failure to prevent data inconsistency. © 2019 TIS Inc. akka-cluster スプリットブレインリゾルバ OSS実装一覧 - Qiita ネットワーク障害によって分裂したクラスターの片方か、もしくは全てを停止させてデータの一貫性が崩れるのを防ぐ 62 / 73

Slide 63

Slide 63 text

Split Brain Solutions for Akka Cluster Split Brain Resolver (Lightbend: Akka Commercial Addons) TanUkkii007/akka-cluster-custom-downing mbilski/akka-reasonable-downing Stop one or all of the split clusters due to network failure to prevent data inconsistency. The stop fire an failover of entities. © 2019 TIS Inc. akka-cluster スプリットブレインリゾルバ OSS実装一覧 - Qiita ネットワーク障害によって分裂したクラスターの片方か、もしくは全てを停止させてデータの一貫性が崩れるのを防ぐこの停止により Entity のフェイルオーバーが引き起こされる 63 / 73

Slide 64

Slide 64 text

Consider downtime due to failure What will happen if Cluster Sharding nodes crashed or the network failure ? Part of users will not be able to use the service. © 2019 TIS Inc. ノードのクラッシュやネットワーク障害によって、一部のユーザーがサービスを利用できなくなる。 64 / 73

Slide 65

Slide 65 text

Downtime indication 25 seconds（for a cluster of 10 nodes） failure detection: 5 seconds stable-after: 10 seconds down-removal-margin: 10 seconds Downtime increases as nodes increase © 2019 TIS Inc. Split Brain Resolver · Akka Commercial Addons ダウンタイムの目安ノードが増えるごとにダウンタイムも増加する。 65 / 73

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Tune recovery time We can tune recovery time by to change Akka Cluter configuration. However, it is trade-off between recovery time and false positives or temporary Split Brain. © 2019 TIS Inc. Akka Cluster の設定を調整し、ダウンタイムを短くすることはできるが、障害の誤検知や一時的な Split Brain とのトレードオフ 67 / 73

Slide 68

Slide 68 text

Data migration Command-side datastore (Cassandra) has no data about past payments. Can not cancel past payments due to it We had to construct Command-side data from payment history which is in RDBMS © 2019 TIS Inc. コマンドサイドには過去の決済データがないため、RDBMSの決済履歴からコマンドサイドのデータを作成する必要があった 68 / 73

Slide 69

Slide 69 text

Summary We could reconstruct a part of CRUD style system to ES + CQRS with Akka It is not necessary to achieve a perfect Reactive System from the beginning © 2019 TIS Inc. CRUDスタイルのシステムの一部分を Akka で再構築。最初から完璧なリアクティブシステムを達成する必要はない。 69 / 73

Slide 70

Slide 70 text

Summary We could reconstruct a part of CRUD style system to ES + CQRS with Akka It is not necessary to achieve a perfect Reactive System from the beginning If you want to challenge it... © 2019 TIS Inc. CRUDスタイルのシステムの一部分を Akka で再構築。最初から完璧なリアクティブシステムを達成する必要はない。 70 / 73