Alpakka with Cloud PubSub

Transcript

1. Alpakkaで Cloud PubSubを ストリーム処理する Fringe81社内Scala勉強会 2018.07.19
2. Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...
3. Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...
4. 簡単にAkka Stream復習
5. 主要モジュールは３つ Source Flow Sink これらを組み合わせる
6. (1 to 3) .map{ i => println(s"A: $i"); i } .map{ i => println(s"B: $i"); i } .map{ i => println(s"C: $i"); i } A:1 A:2 A:3 B:1 B:2 B:3 C:1 C:2 C:3 普通のScalaのコレクション
7. Source(1 to 3) .map{ i => println(s"A: $i"); i } .map{ i => println(s"B: $i"); i } .map{ i => println(s"C: $i"); i } .runWith(Sink.ignore) A:1 A:2 B:1 A:3 B:2 C:1 B:3 C:2 C:3 Akka Stream
8. その他、 排圧制御など色々あるけど今回は割愛
9. Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...
10. Alpakkaいく前に 先にGoogleから提供されている Cloud PubSubのJavaAPIを確認してみる
11. 〜Publish編〜 val msg = ByteString.copyFromUtf8(jsonStr) val pubSubMsg = PubsubMessage.newBuilder.setData(msg).build() val resultF: ApiFuture[String] = publisher.publish(pubSubMsg)
12. 〜Subscribe編〜 このインタフェースを実装する public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); }
13. 〜Subscribe編〜 messageを処理してconsumer.ack()で終わり public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); }
14. public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); } 〜Subscribe編〜 １件のmessageをどう処理するか、という処理モデル
15. https://github.com/openmessaging/specification/blob/master/usecase.md より
16. 一方のAlpakkaによるStream処理
17. https://github.com/openmessaging/specification/blob/master/usecase.md より
18. ではさっそく実装を
19. 〜Subscriber編〜
20. val subscriptionSource: Source[ReceivedMessage, NotUsed] = GooglePubSub.subscribe(projectId, apiKey, email, privateKey, subscription) val ackSink: Sink[AcknowledgeRequest, Future[Done]] = GooglePubSub.acknowledge(projectId, apiKey, email, privateKey, subscription) subscriber用に提供されているのはSourceとSink Soruceは最大1000件のMessageがくる Sinkは入ってきたやつのackを返す
21. val subscriptionSource: Source[ReceivedMessage, NotUsed] = GooglePubSub.subscribe(projectId, apiKey, email, privateKey, subscription) val ackSink: Sink[AcknowledgeRequest, Future[Done]] = GooglePubSub.acknowledge(projectId, apiKey, email, privateKey, subscription) val doneF: Future[Done] = subscriptionSource .grouped(3) .map(msgs => AcknowledgeRequest(msgs.map(_.ackId))) .runWith(ackSink) Flow部分はアプリケーション固有の処理を。 ここでは単純に3件ずつの固まりにしてただack返すだけ
22. 〜Publisher編〜
23. val publishFlow: Flow[PublishRequest, Seq[String], NotUsed] = GooglePubSub.publish(projectId, apiKey, clientEmail, privateKey, topic) publisher用に提供されているのはFlow PublishRequestをinputにpublish実行してくれ その際、採番されたidがoutputとなる
24. val publishFlow: Flow[PublishRequest, Seq[String], NotUsed] = GooglePubSub.publish(projectId, apiKey, clientEmail, privateKey, topic) val source: Source[PublishRequest, NotUsed] = { val msgs = (1 to 10).map { i => PubSubMessage( messageId = i.toString, data = new String(Base64.getEncoder.encode("Hello".getBytes)) )} Source.single(PublishRequest(msgs)) } val publishedMessageIds: Future[Done] = source.via(publishFlow).runWith(Sink.ignore)
25. ところで、JavaAPIだと 一件一件しか取れないはずなのに どうやってCloud Pub/Subから カタマリで取ってるんだろう？
26. def pull(project: String, subscription: String, maybeAccessToken: Option[String], apiKey: String)( implicit as: ActorSystem, materializer: Materializer ): Future[PullResponse] = { import materializer.executionContext val uri: Uri = s"$PubSubGoogleApisHost/v1/projects/$project/subscriptions/$subscription:pull" val request = HttpApi.PullRequest(returnImmediately = true, maxMessages = 1000) PubSub用alpakkaライブラリの中身 Java API使ってると思いこんでたけど Cloud PubSubのWEB API呼んでるだけだった
27. お　わ　り