デブサミ2018 Apache Kafkaの話

Developers Summit2018でのスライド資料です
"Apache Kafkaによるスケーラブル アプリケーション開発"

Transcript

1. Apache KafkaʹΑΔ
   εέʔϥϒϧ
   ΞϓϦέʔγϣϯ։ൃ
   yuuki takezawa 

   Developers Summit 2018 - 2/16
   

   View Slide

2. Agenda
   • What Apache Kafka
   • Kafka Connect / Kafka Streams
   • Kappa Architecture
   • ΞϓϦέʔγϣϯͰ׆༻
   

   View Slide

3. Scalability
   • ن໛ಁաੑ

   ෛՙͷߴ௿ʹ߹ΘͤͯϦιʔεɾϓʔϧΛ

   ֦େɾॖখͰ͖Δ͜ͱ
   • Ґஔಁաੑ

   Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ
   มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ
   • ҟछಁաੑ

   γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜
   ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ
   

   View Slide

4. What is Apache Kafka?
   

   View Slide

5. Message Broker
   

   View Slide

6. Apache Kafka
   • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ
   • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ
   • ϏοάσʔλରԠ
   • ϑΝΠϧγεςϜར༻Ͱɺ

   γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ
   • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ
   • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ

   (Amazon kinesisͱ΄΅ಉ͡)
   

   View Slide

7. Apache Kafka + ZooKeeper Architecture
   

   View Slide

8. Apache Kafka֓ཁ
   • Producer 

   ϝοηʔδ഑৴Λߦ͏

   ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻
   • Consumer 

   ϝοηʔδߪಡΛߦ͏

   ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ

   Ұఆظؒอ؅͞ΕΔ
   • Broker

   KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
   

   View Slide

9. Apache Kafka֓ཁ
   • Topic 

   Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ

   ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ
   • Partition

   ෛՙ෼ࢄ༻్ʹར༻

   ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ

   ͦΕͧΕ͕ॲཧΛߦ͏

   ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
   

   View Slide

10. Example Partition
    

    View Slide

11. Producer αϯϓϧ
    https://github.com/istyle-inc/go-
    example-developers-summit
    

    View Slide

12. import (
    "github.com/confluentinc/confluent-kafka-go/kafka"
    )
    func (kc *KafkaClient) PrepareProducer() {
    p, err := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers": *kc.BootstrapServers,
    "broker.address.family": "v4",
    "queue.buffering.max.messages": "1000",
    "client.id": "testingClient",
    })
    // লུ
    }
    ઀ଓ͢Δ,BGLBΛࢦఆ
    

    View Slide

13. Kafka Connect
    • Kafka Connectͱ͸ɺ

    पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ

    σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ
    • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ

    ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ

    ͕ߦ͑Δ
    • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ

    (java, Scala)
    

    View Slide

14. Example Kafka Connect
    &WFOU4PVSDJOH
    4UBUF4PVSDJOH
    ଞΞϓϦέʔγϣϯ

    $POTVNFS
    ૹ৴಺༰Λͦͷ··

    &MBTUJDTFBSDI΁
    

    View Slide

15. Kafka Connect Sink - Elasticsearch
    

    View Slide

16. bootstrap.servers=192.168.10.10:9092
    key.converter=org.apache.kafka.connect.json.JsonConverter
    value.converter=org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable=false
    value.converter.schemas.enable=false
    internal.key.converter=org.apache.kafka.connect.json.JsonConverter
    internal.value.converter=org.apache.kafka.connect.json.JsonConverter
    internal.key.converter.schemas.enable=false
    internal.value.converter.schemas.enable=false
    offset.storage.file.filename=/tmp/connect.offsets
    offset.flush.interval.ms=10000
    rest.port=8093
    plugin.path=/usr/share/java/
    

    View Slide

17. name=elasticsearch-sink
    connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
    tasks.max=1
    topics=sample.append_employees
    key.ignore=true
    connection.url=http://localhost:9200
    schema.ignore=true
    type.name=kafka-connect
    FMBTUJDTFBSDIDPOOFDU
    ର৅ͷUPQJDΛબ୒
    

    View Slide

18. $ confluent load elasticsearch-sink -d \
    /etc/kafka-connect-elasticsearch/sample-es.properties
    $ connect-standalone \
    -daemon /etc/schema-registry/elasticsearch-connect.properties \
    /etc/kafka-connect-elasticsearch/sample-es.properties
    DPOOFDUΛొ࿥
    సૹ͕։࢝͞Ε·͢
    

    View Slide

19. View Slide

20. Command Query Responsibility Segregation
    

    View Slide

21. Kafka Streams
    • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ
    • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ

    Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ
    • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ
    • ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ

    ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ
    

    View Slide

22. What is Stream?
    

    View Slide

23. ετϦʔϜॲཧ
    • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ

    ετϦʔϜσʔλॲཧͷ໨త
    • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν
    • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ
    • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ
    • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
    

    View Slide

24. Kafka Streams
    • KStream / KTable
    • KStream͸ετϦʔϜͰྲྀΕͯ͘Δσʔλ͕KVͰͦͷ
    ··ྲྀΕͯ͘Δ
    • KStream͸γϯϓϧʹfilterॲཧ΍ɺஔ͖׵͑༻్

    ϩά΍ɺπΠʔτɺλΠϜϥΠϯతͳ΋ͷʹ
    

    View Slide

25. ؆୯ͳαϯϓϧ
    https://github.com/istyle-inc/
    developers-summit-kafka-streams
    

    View Slide

26. def main(args: Array[String]) {
    println("Kafka Streams Sample.")
    val config: Properties = {
    val prop = new Properties()
    prop.load(new java.io.FileInputStream(args(0).toString))
    prop.put(StreamsConfig.APPLICATION_ID_CONFIG, prop.getProperty("sample.app_id"))
    prop.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, prop.getProperty("sample.bootstrap.servers"))
    prop.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass)
    prop.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass)
    // exactly once
    prop.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE)
    prop
    }
    }
    LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG
    αʔϏεશମͰಉ͡BQQ@JE͸࢖Θͳ͍
    ઀ଓ͢ΔTFSWFSΛࢦఆ
    

    View Slide

27. val ft: String = config.getProperty("sample.stream.topic")
    val tt: String = config.getProperty("sample.streamed.topic")
    println("stream topic: from " + ft)
    println("to " + tt)
    val stringSerde: Serde[String] = Serdes.String()
    val builder: StreamsBuilder = new StreamsBuilder
    val rawStream: KStream[String, String] = builder.stream(ft)
    val mapStream: KStream[String, String] = rawStream.mapValues(new ValueMapper[String, String] {
    override def apply(value: String): String = new ElementAppender(value).append()
    })
    mapStream.to(stringSerde, stringSerde, tt)
    val streams: KafkaStreams = new KafkaStreams(builder.build(), config)
    streams.start()
    4USFBNॲཧର৅UPQJD
    4USFBNॲཧޙʹ֨ೲ͢ΔUPQD
    ,4USFBNͰॲཧ
    σʔλͷஔ͖׵͑
    ॲཧޙʹUPQJD֨ೲΛߦ͏
    4USFBNॲཧ໋ྩܧଓతʹ࣮ߦ͞ΕΔ
    

    View Slide

28. ϝοηʔδͷܽଛ
    ૹ৴ࣦഊ
    BDLड৴ࣦഊ
    #SPLFSμ΢ϯ
    ϝοηʔδड৴ࣦഊ
    

    View Slide

29. Exactly Once
    • 0.11͔Β௥Ճ
    • Producer, BrokerؒͷτϥϯβΫγϣϯ

    Broker, ConsumerؒͷτϥϯβΫγϣϯ
    • ਖ਼֬ʹҰ౓͚ͩૹ৴
    

    View Slide

30. ϝοηʔδ఻ୡͷอূ
    • At least once semantics

    ॏෳΛڐՄ
    • At most once semantics

    ܽଛΛڐՄ
    • Exactly once semantics

    ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍

    

    "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 

    https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
    

    View Slide

31. Apache KafkaͰ՝୊ղܾ
    • ྲྀΕͯ͘Δσʔλʹ߹Θͤͯ

    ։ൃऀ͕෼ࢄํ๏Λઃܭ͢Δ͜ͱ͕Ͱ͖Δ
    • ϩάͷτϥϯβΫγϣϯ

    ESɾCQRSͰΞϓϦέʔγϣϯʹऔΓࠐΉ͜ͱ͕Մೳ
    • Kafka ConnectɺKafka Streams͕ڧྗ

    • HadoopΤίγεςϜͰ਌࿨ੑ
    

    View Slide

32. ڊେԽ͢ΔΞϓϦέʔγϣϯʹ
    

    View Slide

33. γϯϓϧͳΞϓϦέʔγϣϯߏ੒
    

    View Slide

34. γϯϓϧͳΞϓϦέʔγϣϯߏ੒
    

    View Slide

35. ڊେͳΞϓϦέʔγϣϯ΁
    

    View Slide

36. ෳࡶʹͳΔΞϓϦέʔγϣϯ
    • σʔλಉظͷෳࡶԽ

    ϦΞϧλΠϜͰσʔλ͕ཉ͍͠

    ΠϯσοΫεߋ৽͕ఆظ࣮ߦ
    • ͦͷଞ

    Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ
    • ෳࡶԽ͢Δ࣮૷ίʔυ
    

    View Slide

37. എܠ
    • େ͖͘ෳࡶԽ͢ΔΞϓϦέʔγϣϯͷ੹຿

    -> ݶΒΕͨػೳͷΈఏڙ͢Δ 

    -> ϚΠΫϩαʔϏεΑΓʹ͢Δඞཁ͕͋ͬͨ
    • ೗Կʹεέʔϧ͍͔ͤͯ͘͞

    -> ͍͍ϋʔυ΢ΣΞͰؤுΔʹ΋ݶք͕͋Δ
    • ߴՄ༻ɾো֐଱ੑ

    -> σʔλଛࣦ୲อɺϦτϥΠͳͲ͕Մೳͳ΋ͷ͕ʂ
    

    View Slide

38. ௨஌ػೳʹಋೖ
    • ͳʹͳʹ͞Μ͕Ͳ͜ͰԿ͠·ͨ͠

    ԿϙΠϯτ֫ಘ͠·ͨ͠ʂ etc
    • αʔϏεԣஅͰ࢖ΘΕΔ
    • ਺100ສ෼ͷEvent (ଟ͍ͱ͖)
    • σʔλՃ޻ʹRDBMSͱ૊Έ߹ΘͤΔඞཁ͕͋Γɺ

    αʔϏεʹΑͬͯ͸ෳࡶʹͳΔ΋ͷ΋͋ΔͨΊɺ

    Consumerͷෛՙ෼ࢄΛ༰қʹ͍ͨ͠
    • ϝοηʔδଛࣦΛͳΔ΂͘๷͍͗ͨ
    

    View Slide

39. ௨஌ػೳ
    1SPEVDFSந৅Խ
    ϥΠϒϥϦఏڙ
    ௨஌ొ࿥"1*ఏڙ
    UPQJDʹ߹Θͤͯ
    1BSUJUJPO/
    ଞͷαʔϏεͰSBCCJUNRΛར༻
    ͍ͯ͠ΔͨΊɺSBCCJUNR΁
    ,BGLB$POOFDUʹมߋ༧ఆ
    $BTTBOESBΫϥελ
    

    View Slide

40. ಋೖޙ
    • Ұ೥ؒͷ͏ͪ໨ཱͬͨো֐͸ͳ͠

    • ϝοηʔδ֬ೝ͕༰қͰɺૄ௨֬ೝͳͲॏๅ

    • KafkaɺCassandraͷ૊Έ߹ΘͤͰࡹ͚͗ͯ͢͠·͍ɺ

    ͍ͭ΋εΧεΧ

    • rabbitmqͷόοΫΞοϓͱͯ͠΋Քಇ

    *ࣾ಺Ͱrabbitmqར༻αʔϏεͷํ͕ଟ͍ͨΊ
    

    View Slide

41. View Slide

42. Kafka Connect Source - Rabbitmq
    

    View Slide

43. name=named-kafka-connect-rabbitmq
    tasks.max=1
    connector.class=com.github.jcustenborder.kafka.connect.rabbitmq.RabbitMQSour
    ceConnector
    rabbitmq.prefetch.count=500
    rabbitmq.automatic.recovery.enabled=true
    rabbitmq.network.recovery.interval.ms=10000
    rabbitmq.topology.recovery.enabled=true
    rabbitmq.queue=action_api.create_object
    kafka.topic=rabbitmq.source_topic
    rabbitmq.host=rabbitmqhost.local
    rabbitmq.password=password
    rabbitmq.username=username
    

    View Slide

44. $ confluent load named-kafka-connect-rabbitmq -d \
    /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties
    $ connect-standalone \
    -daemon /etc/schema-registry/connect-json-standalone.properties \
    /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties
    

    View Slide

45. RabbitMQ -> Kafka
    ,BGLB$POOFDUͰ
    ٵ্͍͛
    ,BGLB4USFBNTͰ
    UPQJDৼΓ෼͚
    LBGLBDPOOFDUIEGTͰ
    IEGTʹग़ྗ
    

    View Slide

46. Lambda Architecture
    

    View Slide

47. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊
    • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ
    ཧ͕ऴΘΒͳ͍
    • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ

    σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠
    • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠
    • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠
    • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔
    

    View Slide

48. BigData΁ͷΞϓϩʔν
    • σʔλͦͷ΋ͷͷू໿
    • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻
    ҙ͢Δ
    • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ

    ߴ଎ͳMessageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ
    

    View Slide

49. Lambda Architecture
    • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒
    • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳ
    ͲΛ୲౰͢Δ -> Hadoop(MapReduce), Spark
    • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ

    Hive, HBase, ElephantDB, Splout SQL, pipelineDB…
    • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚

    Spark, Storm, Kafka, Cassandra etc..
    • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫

    αʔϏεʹΑͬͯ͸೉қ౓͕ߴ͍ɾɾʂ
    

    View Slide

50. Lambda Architecture
    

    View Slide

51. Kappa Architecture
    

    View Slide

52. KappaΞʔΩςΫνϟ
    

    View Slide

53. KappaΞʔΩςΫνϟ
    

    View Slide

54. def main(args: Array[String]) {
    val kafkaParams = Map[String, Object](
    "bootstrap.servers" -> "localhost:9092",
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[StringDeserializer],
    "group.id" -> "kafka_builderscon_stream",
    "auto.offset.reset" -> "latest",
    "enable.auto.commit" -> (false: java.lang.Boolean)
    )
    //
    val spark = SparkSession
    .builder
    .master("local[*]")
    .appName("buildersconSmaple")
    .config("spark.cassandra.connection.host", "192.168.10.10")
    .getOrCreate()
    val streamingContext = new StreamingContext(spark.sparkContext, Seconds(5))
    LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG
    ,BGLB઀ଓ৘ใ
    ࠷৽ͷྲྀΕͯདྷͨσʔλΛ࢖ͬͯৗʹॲཧ
    4QBSL"QQ໊
    4USFBNJOHॲཧ݁ՌΛ$BTTBOESBʹ
    ඵ͓͖ʹશσʔλΛॻ͖ࠐΉ
    

    View Slide

55. streamingContext.checkpoint("/tmp/")
    val topics = Array("message-topic")
    val stream = KafkaUtils.createDirectStream[String, String](
    streamingContext,
    PreferConsistent,
    Subscribe[String, String](topics, kafkaParams)
    )
    val pairs = stream.map(record => (record.value, 1))
    val count = pairs.updateStateByKey(updateFunc)
    ॲཧσʔλͷܦաΛॻ͖ࠐΉ
    ࢮΜͰ΋͔͜͜Β΍Γ௚͢
    ,BGLB5PQJDͷσʔλΛ
    4QBSLͷ4USFBNʹྲྀ͠ࠐΉ
    

    View Slide

56. count.foreachRDD((rdd, time) => {
    val count = rdd.count()
    if (count > 0) {
    rdd.map(record => ("spark", streamMessageParse(record._1.toString).message, record._2))
    .saveToCassandra("builderscon", "counter", SomeColumns("stream", "message", "counter"))
    }
    })
    count.print()
    streamingContext.start()
    streamingContext.awaitTermination()
    }
    ྲྀΕͯ͘Δ3%%ͷॲཧ։࢝
    ूܭ݁ՌΛ$BTTBOESBͷΧϥϜʹNBQ
    ͯ͠ॻ͖ࠐΉ
    

    View Slide

57. for Application
    

    View Slide

58. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • ProducerͰόϦσʔγϣϯ

    -> ࠷௿ݶͷܕकΔ

    -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍

    • ConsumerͰόϦσʔγϣϯ

    ड৴ͨ͠ϝοηʔδΛௐ΂ɺ

    ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ
    

    View Slide

59. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • Kafka Streamͱ͍͏બ୒ࢶ

    java·ͨ͸ScalaͰ࣮૷Մೳ

    -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍

    • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏

    -> ϚΠΫϩαʔϏεͰ͋Ε͹
    

    View Slide

60. PipelineDBͱ͍͏બ୒ࢶ
    • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ

    ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠

    • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍

    • PostgreSQL Compatible

    • PDOͰ઀ଓՄೳʂ
    

    View Slide

61. PipelineDBͱ͍͏બ୒ࢶ
    CREATE EXTENSION pipeline_kafka;
    SELECT pipeline_kafka.add_broker(‘localhost:9092');
    CREATE STREAM logs_stream (payload json);
    CREATE CONTINUOUS VIEW message_count
    AS SELECT COUNT(*) FROM logs_stream;
    

    View Slide

62. KSQL
    • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ

    SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ
    • ݱࡏDeveloper Preview
    • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢
    

    View Slide

63. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ
    • PrestoΛར༻͠ɺ

    RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ

    RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ

    KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλֻ͚߹Θͤ
    

    View Slide

64. for Prestodb
    • ܕఆٛ etc/kafka/table໊.json
    • ઀ଓ৘ใ “etc/catalog/kafka_tests.properties”
    • prestoͷΧλϩάʹ઀ଓ৘ใΛهड़
    • topic໊͸ hoge.fuga ܗࣜͱ͢Δ
    

    View Slide

65. {
    "tableName": “action",
    "schemaName": "analyze",
    "topicName": "analyze.action",
    "message": {
    "dataFormat": "json",
    "fields": [
    {
    "name": "uuid",
    "mapping": "uuid",
    "type": "VARCHAR"
    },
    {
    "name": "uri",
    "mapping": "uri",
    "type": "VARCHAR"
    },
    {
    "name": "name",
    "mapping": "name",
    "type": "VARCHAR"
    }
    ]
    }
    LBGLBͷUPQJD৘ใΛهࡌ
    ΧϥϜΛఆٛϓϦϛςΟϒͳܕʹม׵
    

    View Slide

66. connector.name=kafka
    kafka.nodes=127.0.0.1:9092
    kafka.table-names=analyze.action
    kafka.hide-internal-columns=false
    DPOODUPS໊͸LBGLB
    LBGLBͷΫϥελʔΛࢦఆ
    LBGLBͷUPQJDΛࢦఆ
    LBGLBͷγεςϜΧϥϜ༗ແ
    

    View Slide

67. SELECT redttt._key, redttt._value, test_id,
    test_name, created_at, uri, uuid
    FROM my_tests.testing.tests AS myttt
    INNER JOIN red_tests.test.string AS redttt ON
    redttt._key = myttt.test_name
    INNER JOIN kafka_tests.analyze.action AS kafkataa
    ON kafkataa.name = myttt.test_name
    WHERE myttt.test_name = '{$name}' LIMIT 1";
    

    View Slide

68. Apache Kafkaͷ࢝Ίํ
    • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ
    • Confluent͔Βμ΢ϯϩʔυ
    • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ

    Confluentར༻͕Φεεϝ
    • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢
    

    View Slide

69. Apache Kafka GUI
    • Cluster؅ཧ 

    https://github.com/yahoo/kafka-manager
    • Message؅ཧ 

    https://github.com/landoop/kafka-topics-ui

    https://github.com/ldaniels528/trifecta

    https://github.com/Landoop/kafka-topics-ui

    https://github.com/Landoop/kafka-connect-ui

    ͳͲ
    

    View Slide

70. ·ͱΊ
    • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ
    ͷ໰୊ղܾͷҰͭ
    • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ
    • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑
    • ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ
    

    View Slide