Slide 1

Slide 1 text

Alpakkaで Cloud PubSubを ストリーム処理する Fringe81社内Scala勉強会 2018.07.19

Slide 2

Slide 2 text

Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...

Slide 3

Slide 3 text

Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...

Slide 4

Slide 4 text

簡単にAkka Stream復習

Slide 5

Slide 5 text

主要モジュールは3つ Source Flow Sink これらを組み合わせる

Slide 6

Slide 6 text

(1 to 3) .map{ i => println(s"A: $i"); i } .map{ i => println(s"B: $i"); i } .map{ i => println(s"C: $i"); i } A:1 A:2 A:3 B:1 B:2 B:3 C:1 C:2 C:3 普通のScalaのコレクション

Slide 7

Slide 7 text

Source(1 to 3) .map{ i => println(s"A: $i"); i } .map{ i => println(s"B: $i"); i } .map{ i => println(s"C: $i"); i } .runWith(Sink.ignore) A:1 A:2 B:1 A:3 B:2 C:1 B:3 C:2 C:3 Akka Stream

Slide 8

Slide 8 text

その他、 排圧制御など色々あるけど今回は割愛

Slide 9

Slide 9 text

Alpakka AkkaStream Connector for integration! AMQP, MQTT, SQS, SNS, Kinesis, Cloud PubSub, Azure Event Hub, Firebase cloud messaging, S3, Kafka, MongoDB, HBase, Slick, HTTP, TCP, etc,...

Slide 10

Slide 10 text

Alpakkaいく前に 先にGoogleから提供されている Cloud PubSubのJavaAPIを確認してみる

Slide 11

Slide 11 text

〜Publish編〜 val msg = ByteString.copyFromUtf8(jsonStr) val pubSubMsg = PubsubMessage.newBuilder.setData(msg).build() val resultF: ApiFuture[String] = publisher.publish(pubSubMsg)

Slide 12

Slide 12 text

〜Subscribe編〜 このインタフェースを実装する public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); }

Slide 13

Slide 13 text

〜Subscribe編〜 messageを処理してconsumer.ack()で終わり public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); }

Slide 14

Slide 14 text

public interface MessageReceiver { void receiveMessage( final PubsubMessage message, final AckReplyConsumer consumer ); } 〜Subscribe編〜 1件のmessageをどう処理するか、という処理モデル

Slide 15

Slide 15 text

https://github.com/openmessaging/specification/blob/master/usecase.md より

Slide 16

Slide 16 text

一方のAlpakkaによるStream処理

Slide 17

Slide 17 text

https://github.com/openmessaging/specification/blob/master/usecase.md より

Slide 18

Slide 18 text

ではさっそく実装を

Slide 19

Slide 19 text

〜Subscriber編〜

Slide 20

Slide 20 text

val subscriptionSource: Source[ReceivedMessage, NotUsed] = GooglePubSub.subscribe(projectId, apiKey, email, privateKey, subscription) val ackSink: Sink[AcknowledgeRequest, Future[Done]] = GooglePubSub.acknowledge(projectId, apiKey, email, privateKey, subscription) subscriber用に提供されているのはSourceとSink Soruceは最大1000件のMessageがくる Sinkは入ってきたやつのackを返す

Slide 21

Slide 21 text

val subscriptionSource: Source[ReceivedMessage, NotUsed] = GooglePubSub.subscribe(projectId, apiKey, email, privateKey, subscription) val ackSink: Sink[AcknowledgeRequest, Future[Done]] = GooglePubSub.acknowledge(projectId, apiKey, email, privateKey, subscription) val doneF: Future[Done] = subscriptionSource .grouped(3) .map(msgs => AcknowledgeRequest(msgs.map(_.ackId))) .runWith(ackSink) Flow部分はアプリケーション固有の処理を。 ここでは単純に3件ずつの固まりにしてただack返すだけ

Slide 22

Slide 22 text

〜Publisher編〜

Slide 23

Slide 23 text

val publishFlow: Flow[PublishRequest, Seq[String], NotUsed] = GooglePubSub.publish(projectId, apiKey, clientEmail, privateKey, topic) publisher用に提供されているのはFlow PublishRequestをinputにpublish実行してくれ その際、採番されたidがoutputとなる

Slide 24

Slide 24 text

val publishFlow: Flow[PublishRequest, Seq[String], NotUsed] = GooglePubSub.publish(projectId, apiKey, clientEmail, privateKey, topic) val source: Source[PublishRequest, NotUsed] = { val msgs = (1 to 10).map { i => PubSubMessage( messageId = i.toString, data = new String(Base64.getEncoder.encode("Hello".getBytes)) )} Source.single(PublishRequest(msgs)) } val publishedMessageIds: Future[Done] = source.via(publishFlow).runWith(Sink.ignore)

Slide 25

Slide 25 text

ところで、JavaAPIだと 一件一件しか取れないはずなのに どうやってCloud Pub/Subから カタマリで取ってるんだろう?

Slide 26

Slide 26 text

def pull(project: String, subscription: String, maybeAccessToken: Option[String], apiKey: String)( implicit as: ActorSystem, materializer: Materializer ): Future[PullResponse] = { import materializer.executionContext val uri: Uri = s"$PubSubGoogleApisHost/v1/projects/$project/subscriptions/$subscription:pull" val request = HttpApi.PullRequest(returnImmediately = true, maxMessages = 1000) PubSub用alpakkaライブラリの中身 Java API使ってると思いこんでたけど Cloud PubSubのWEB API呼んでるだけだった

Slide 27

Slide 27 text

お わ り