Upgrade to Pro — share decks privately, control downloads, hide ads and more …

デブサミ2018 Apache Kafkaの話

yuuki takezawa
February 02, 2018

デブサミ2018 Apache Kafkaの話

Developers Summit2018でのスライド資料です
"Apache Kafkaによるスケーラブル アプリケーション開発"

yuuki takezawa

February 02, 2018
Tweet

More Decks by yuuki takezawa

Other Decks in Technology

Transcript

  1. Apache KafkaʹΑΔ
    εέʔϥϒϧ
    ΞϓϦέʔγϣϯ։ൃ
    yuuki takezawa 

    Developers Summit 2018 - 2/16

    View full-size slide

  2. Agenda
    • What Apache Kafka
    • Kafka Connect / Kafka Streams
    • Kappa Architecture
    • ΞϓϦέʔγϣϯͰ׆༻

    View full-size slide

  3. Scalability
    • ن໛ಁաੑ

    ෛՙͷߴ௿ʹ߹ΘͤͯϦιʔεɾϓʔϧΛ

    ֦େɾॖখͰ͖Δ͜ͱ
    • Ґஔಁաੑ

    Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ
    มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ
    • ҟछಁաੑ

    γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜
    ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ

    View full-size slide

  4. What is Apache Kafka?

    View full-size slide

  5. Message Broker

    View full-size slide

  6. Apache Kafka
    • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ
    • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ
    • ϏοάσʔλରԠ
    • ϑΝΠϧγεςϜར༻Ͱɺ

    γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ
    • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ
    • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ

    (Amazon kinesisͱ΄΅ಉ͡)

    View full-size slide

  7. Apache Kafka + ZooKeeper Architecture

    View full-size slide

  8. Apache Kafka֓ཁ
    • Producer 

    ϝοηʔδ഑৴Λߦ͏

    ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻
    • Consumer 

    ϝοηʔδߪಡΛߦ͏

    ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ

    Ұఆظؒอ؅͞ΕΔ
    • Broker

    KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ

    View full-size slide

  9. Apache Kafka֓ཁ
    • Topic 

    Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ

    ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ
    • Partition

    ෛՙ෼ࢄ༻్ʹར༻

    ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ

    ͦΕͧΕ͕ॲཧΛߦ͏

    ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏

    View full-size slide

  10. Example Partition

    View full-size slide

  11. Producer αϯϓϧ
    https://github.com/istyle-inc/go-
    example-developers-summit

    View full-size slide

  12. import (
    "github.com/confluentinc/confluent-kafka-go/kafka"
    )
    func (kc *KafkaClient) PrepareProducer() {
    p, err := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers": *kc.BootstrapServers,
    "broker.address.family": "v4",
    "queue.buffering.max.messages": "1000",
    "client.id": "testingClient",
    })
    // লུ
    }
    ઀ଓ͢Δ,BGLBΛࢦఆ

    View full-size slide

  13. Kafka Connect
    • Kafka Connectͱ͸ɺ

    पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ

    σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ
    • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ

    ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ

    ͕ߦ͑Δ
    • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ

    (java, Scala)

    View full-size slide

  14. Example Kafka Connect
    &WFOU4PVSDJOH
    4UBUF4PVSDJOH
    ଞΞϓϦέʔγϣϯ

    $POTVNFS
    ૹ৴಺༰Λͦͷ··

    &MBTUJDTFBSDI΁

    View full-size slide

  15. Kafka Connect Sink - Elasticsearch

    View full-size slide

  16. bootstrap.servers=192.168.10.10:9092
    key.converter=org.apache.kafka.connect.json.JsonConverter
    value.converter=org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable=false
    value.converter.schemas.enable=false
    internal.key.converter=org.apache.kafka.connect.json.JsonConverter
    internal.value.converter=org.apache.kafka.connect.json.JsonConverter
    internal.key.converter.schemas.enable=false
    internal.value.converter.schemas.enable=false
    offset.storage.file.filename=/tmp/connect.offsets
    offset.flush.interval.ms=10000
    rest.port=8093
    plugin.path=/usr/share/java/

    View full-size slide

  17. name=elasticsearch-sink
    connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
    tasks.max=1
    topics=sample.append_employees
    key.ignore=true
    connection.url=http://localhost:9200
    schema.ignore=true
    type.name=kafka-connect
    FMBTUJDTFBSDIDPOOFDU
    ର৅ͷUPQJDΛબ୒

    View full-size slide

  18. $ confluent load elasticsearch-sink -d \
    /etc/kafka-connect-elasticsearch/sample-es.properties
    $ connect-standalone \
    -daemon /etc/schema-registry/elasticsearch-connect.properties \
    /etc/kafka-connect-elasticsearch/sample-es.properties
    DPOOFDUΛొ࿥
    సૹ͕։࢝͞Ε·͢

    View full-size slide

  19. Command Query Responsibility Segregation

    View full-size slide

  20. Kafka Streams
    • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ
    • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ

    Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ
    • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ
    • ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ

    ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ

    View full-size slide

  21. What is Stream?

    View full-size slide

  22. ετϦʔϜॲཧ
    • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ

    ετϦʔϜσʔλॲཧͷ໨త
    • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν
    • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ
    • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ
    • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ

    View full-size slide

  23. Kafka Streams
    • KStream / KTable
    • KStream͸ετϦʔϜͰྲྀΕͯ͘Δσʔλ͕KVͰͦͷ
    ··ྲྀΕͯ͘Δ
    • KStream͸γϯϓϧʹfilterॲཧ΍ɺஔ͖׵͑༻్

    ϩά΍ɺπΠʔτɺλΠϜϥΠϯతͳ΋ͷʹ

    View full-size slide

  24. ؆୯ͳαϯϓϧ
    https://github.com/istyle-inc/
    developers-summit-kafka-streams

    View full-size slide

  25. def main(args: Array[String]) {
    println("Kafka Streams Sample.")
    val config: Properties = {
    val prop = new Properties()
    prop.load(new java.io.FileInputStream(args(0).toString))
    prop.put(StreamsConfig.APPLICATION_ID_CONFIG, prop.getProperty("sample.app_id"))
    prop.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, prop.getProperty("sample.bootstrap.servers"))
    prop.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass)
    prop.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass)
    // exactly once
    prop.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE)
    prop
    }
    }
    LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG
    αʔϏεશମͰಉ͡BQQ@JE͸࢖Θͳ͍
    ઀ଓ͢ΔTFSWFSΛࢦఆ

    View full-size slide

  26. val ft: String = config.getProperty("sample.stream.topic")
    val tt: String = config.getProperty("sample.streamed.topic")
    println("stream topic: from " + ft)
    println("to " + tt)
    val stringSerde: Serde[String] = Serdes.String()
    val builder: StreamsBuilder = new StreamsBuilder
    val rawStream: KStream[String, String] = builder.stream(ft)
    val mapStream: KStream[String, String] = rawStream.mapValues(new ValueMapper[String, String] {
    override def apply(value: String): String = new ElementAppender(value).append()
    })
    mapStream.to(stringSerde, stringSerde, tt)
    val streams: KafkaStreams = new KafkaStreams(builder.build(), config)
    streams.start()
    4USFBNॲཧର৅UPQJD
    4USFBNॲཧޙʹ֨ೲ͢ΔUPQD
    ,4USFBNͰॲཧ
    σʔλͷஔ͖׵͑
    ॲཧޙʹUPQJD֨ೲΛߦ͏
    4USFBNॲཧ໋ྩܧଓతʹ࣮ߦ͞ΕΔ

    View full-size slide

  27. ϝοηʔδͷܽଛ
    ૹ৴ࣦഊ
    BDLड৴ࣦഊ
    #SPLFSμ΢ϯ
    ϝοηʔδड৴ࣦഊ

    View full-size slide

  28. Exactly Once
    • 0.11͔Β௥Ճ
    • Producer, BrokerؒͷτϥϯβΫγϣϯ

    Broker, ConsumerؒͷτϥϯβΫγϣϯ
    • ਖ਼֬ʹҰ౓͚ͩૹ৴

    View full-size slide

  29. ϝοηʔδ఻ୡͷอূ
    • At least once semantics

    ॏෳΛڐՄ
    • At most once semantics

    ܽଛΛڐՄ
    • Exactly once semantics

    ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍


    "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 

    https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর

    View full-size slide

  30. Apache KafkaͰ՝୊ղܾ
    • ྲྀΕͯ͘Δσʔλʹ߹Θͤͯ

    ։ൃऀ͕෼ࢄํ๏Λઃܭ͢Δ͜ͱ͕Ͱ͖Δ
    • ϩάͷτϥϯβΫγϣϯ

    ESɾCQRSͰΞϓϦέʔγϣϯʹऔΓࠐΉ͜ͱ͕Մೳ
    • Kafka ConnectɺKafka Streams͕ڧྗ

    • HadoopΤίγεςϜͰ਌࿨ੑ

    View full-size slide

  31. ڊେԽ͢ΔΞϓϦέʔγϣϯʹ

    View full-size slide

  32. γϯϓϧͳΞϓϦέʔγϣϯߏ੒

    View full-size slide

  33. γϯϓϧͳΞϓϦέʔγϣϯߏ੒

    View full-size slide

  34. ڊେͳΞϓϦέʔγϣϯ΁

    View full-size slide

  35. ෳࡶʹͳΔΞϓϦέʔγϣϯ
    • σʔλಉظͷෳࡶԽ

    ϦΞϧλΠϜͰσʔλ͕ཉ͍͠

    ΠϯσοΫεߋ৽͕ఆظ࣮ߦ
    • ͦͷଞ

    Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ
    • ෳࡶԽ͢Δ࣮૷ίʔυ

    View full-size slide

  36. എܠ
    • େ͖͘ෳࡶԽ͢ΔΞϓϦέʔγϣϯͷ੹຿

    -> ݶΒΕͨػೳͷΈఏڙ͢Δ 

    -> ϚΠΫϩαʔϏεΑΓʹ͢Δඞཁ͕͋ͬͨ
    • ೗Կʹεέʔϧ͍͔ͤͯ͘͞

    -> ͍͍ϋʔυ΢ΣΞͰؤுΔʹ΋ݶք͕͋Δ
    • ߴՄ༻ɾো֐଱ੑ

    -> σʔλଛࣦ୲อɺϦτϥΠͳͲ͕Մೳͳ΋ͷ͕ʂ

    View full-size slide

  37. ௨஌ػೳʹಋೖ
    • ͳʹͳʹ͞Μ͕Ͳ͜ͰԿ͠·ͨ͠

    ԿϙΠϯτ֫ಘ͠·ͨ͠ʂ etc
    • αʔϏεԣஅͰ࢖ΘΕΔ
    • ਺100ສ෼ͷEvent (ଟ͍ͱ͖)
    • σʔλՃ޻ʹRDBMSͱ૊Έ߹ΘͤΔඞཁ͕͋Γɺ

    αʔϏεʹΑͬͯ͸ෳࡶʹͳΔ΋ͷ΋͋ΔͨΊɺ

    Consumerͷෛՙ෼ࢄΛ༰қʹ͍ͨ͠
    • ϝοηʔδଛࣦΛͳΔ΂͘๷͍͗ͨ

    View full-size slide

  38. ௨஌ػೳ
    1SPEVDFSந৅Խ
    ϥΠϒϥϦఏڙ
    ௨஌ొ࿥"1*ఏڙ
    UPQJDʹ߹Θͤͯ
    1BSUJUJPO/
    ଞͷαʔϏεͰSBCCJUNRΛར༻
    ͍ͯ͠ΔͨΊɺSBCCJUNR΁
    ,BGLB$POOFDUʹมߋ༧ఆ
    $BTTBOESBΫϥελ

    View full-size slide

  39. ಋೖޙ
    • Ұ೥ؒͷ͏ͪ໨ཱͬͨো֐͸ͳ͠

    • ϝοηʔδ֬ೝ͕༰қͰɺૄ௨֬ೝͳͲॏๅ

    • KafkaɺCassandraͷ૊Έ߹ΘͤͰࡹ͚͗ͯ͢͠·͍ɺ

    ͍ͭ΋εΧεΧ

    • rabbitmqͷόοΫΞοϓͱͯ͠΋Քಇ

    *ࣾ಺Ͱrabbitmqར༻αʔϏεͷํ͕ଟ͍ͨΊ

    View full-size slide

  40. Kafka Connect Source - Rabbitmq

    View full-size slide

  41. name=named-kafka-connect-rabbitmq
    tasks.max=1
    connector.class=com.github.jcustenborder.kafka.connect.rabbitmq.RabbitMQSour
    ceConnector
    rabbitmq.prefetch.count=500
    rabbitmq.automatic.recovery.enabled=true
    rabbitmq.network.recovery.interval.ms=10000
    rabbitmq.topology.recovery.enabled=true
    rabbitmq.queue=action_api.create_object
    kafka.topic=rabbitmq.source_topic
    rabbitmq.host=rabbitmqhost.local
    rabbitmq.password=password
    rabbitmq.username=username

    View full-size slide

  42. $ confluent load named-kafka-connect-rabbitmq -d \
    /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties
    $ connect-standalone \
    -daemon /etc/schema-registry/connect-json-standalone.properties \
    /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties

    View full-size slide

  43. RabbitMQ -> Kafka
    ,BGLB$POOFDUͰ
    ٵ্͍͛
    ,BGLB4USFBNTͰ
    UPQJDৼΓ෼͚
    LBGLBDPOOFDUIEGTͰ
    IEGTʹग़ྗ

    View full-size slide

  44. Lambda Architecture

    View full-size slide

  45. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊
    • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ
    ཧ͕ऴΘΒͳ͍
    • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ

    σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠
    • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠
    • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠
    • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔

    View full-size slide

  46. BigData΁ͷΞϓϩʔν
    • σʔλͦͷ΋ͷͷू໿
    • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻
    ҙ͢Δ
    • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ

    ߴ଎ͳMessageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ

    View full-size slide

  47. Lambda Architecture
    • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒
    • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳ
    ͲΛ୲౰͢Δ -> Hadoop(MapReduce), Spark
    • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ

    Hive, HBase, ElephantDB, Splout SQL, pipelineDB…
    • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚

    Spark, Storm, Kafka, Cassandra etc..
    • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫

    αʔϏεʹΑͬͯ͸೉қ౓͕ߴ͍ɾɾʂ

    View full-size slide

  48. Lambda Architecture

    View full-size slide

  49. Kappa Architecture

    View full-size slide

  50. KappaΞʔΩςΫνϟ

    View full-size slide

  51. KappaΞʔΩςΫνϟ

    View full-size slide

  52. def main(args: Array[String]) {
    val kafkaParams = Map[String, Object](
    "bootstrap.servers" -> "localhost:9092",
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[StringDeserializer],
    "group.id" -> "kafka_builderscon_stream",
    "auto.offset.reset" -> "latest",
    "enable.auto.commit" -> (false: java.lang.Boolean)
    )
    //
    val spark = SparkSession
    .builder
    .master("local[*]")
    .appName("buildersconSmaple")
    .config("spark.cassandra.connection.host", "192.168.10.10")
    .getOrCreate()
    val streamingContext = new StreamingContext(spark.sparkContext, Seconds(5))
    LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG
    ,BGLB઀ଓ৘ใ
    ࠷৽ͷྲྀΕͯདྷͨσʔλΛ࢖ͬͯৗʹॲཧ
    4QBSL"QQ໊
    4USFBNJOHॲཧ݁ՌΛ$BTTBOESBʹ
    ඵ͓͖ʹશσʔλΛॻ͖ࠐΉ

    View full-size slide

  53. streamingContext.checkpoint("/tmp/")
    val topics = Array("message-topic")
    val stream = KafkaUtils.createDirectStream[String, String](
    streamingContext,
    PreferConsistent,
    Subscribe[String, String](topics, kafkaParams)
    )
    val pairs = stream.map(record => (record.value, 1))
    val count = pairs.updateStateByKey(updateFunc)
    ॲཧσʔλͷܦաΛॻ͖ࠐΉ
    ࢮΜͰ΋͔͜͜Β΍Γ௚͢
    ,BGLB5PQJDͷσʔλΛ
    4QBSLͷ4USFBNʹྲྀ͠ࠐΉ

    View full-size slide

  54. count.foreachRDD((rdd, time) => {
    val count = rdd.count()
    if (count > 0) {
    rdd.map(record => ("spark", streamMessageParse(record._1.toString).message, record._2))
    .saveToCassandra("builderscon", "counter", SomeColumns("stream", "message", "counter"))
    }
    })
    count.print()
    streamingContext.start()
    streamingContext.awaitTermination()
    }
    ྲྀΕͯ͘Δ3%%ͷॲཧ։࢝
    ूܭ݁ՌΛ$BTTBOESBͷΧϥϜʹNBQ
    ͯ͠ॻ͖ࠐΉ

    View full-size slide

  55. for Application

    View full-size slide

  56. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • ProducerͰόϦσʔγϣϯ

    -> ࠷௿ݶͷܕकΔ

    -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍

    • ConsumerͰόϦσʔγϣϯ

    ड৴ͨ͠ϝοηʔδΛௐ΂ɺ

    ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ

    View full-size slide

  57. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • Kafka Streamͱ͍͏બ୒ࢶ

    java·ͨ͸ScalaͰ࣮૷Մೳ

    -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍

    • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏

    -> ϚΠΫϩαʔϏεͰ͋Ε͹

    View full-size slide

  58. PipelineDBͱ͍͏બ୒ࢶ
    • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ

    ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠

    • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍

    • PostgreSQL Compatible

    • PDOͰ઀ଓՄೳʂ

    View full-size slide

  59. PipelineDBͱ͍͏બ୒ࢶ
    CREATE EXTENSION pipeline_kafka;
    SELECT pipeline_kafka.add_broker(‘localhost:9092');
    CREATE STREAM logs_stream (payload json);
    CREATE CONTINUOUS VIEW message_count
    AS SELECT COUNT(*) FROM logs_stream;

    View full-size slide

  60. KSQL
    • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ

    SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ
    • ݱࡏDeveloper Preview
    • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢

    View full-size slide

  61. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ
    • PrestoΛར༻͠ɺ

    RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ

    RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ

    KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλֻ͚߹Θͤ

    View full-size slide

  62. for Prestodb
    • ܕఆٛ etc/kafka/table໊.json
    • ઀ଓ৘ใ “etc/catalog/kafka_tests.properties”
    • prestoͷΧλϩάʹ઀ଓ৘ใΛهड़
    • topic໊͸ hoge.fuga ܗࣜͱ͢Δ

    View full-size slide

  63. {
    "tableName": “action",
    "schemaName": "analyze",
    "topicName": "analyze.action",
    "message": {
    "dataFormat": "json",
    "fields": [
    {
    "name": "uuid",
    "mapping": "uuid",
    "type": "VARCHAR"
    },
    {
    "name": "uri",
    "mapping": "uri",
    "type": "VARCHAR"
    },
    {
    "name": "name",
    "mapping": "name",
    "type": "VARCHAR"
    }
    ]
    }
    LBGLBͷUPQJD৘ใΛهࡌ
    ΧϥϜΛఆٛϓϦϛςΟϒͳܕʹม׵

    View full-size slide

  64. connector.name=kafka
    kafka.nodes=127.0.0.1:9092
    kafka.table-names=analyze.action
    kafka.hide-internal-columns=false
    DPOODUPS໊͸LBGLB
    LBGLBͷΫϥελʔΛࢦఆ
    LBGLBͷUPQJDΛࢦఆ
    LBGLBͷγεςϜΧϥϜ༗ແ

    View full-size slide

  65. SELECT redttt._key, redttt._value, test_id,
    test_name, created_at, uri, uuid
    FROM my_tests.testing.tests AS myttt
    INNER JOIN red_tests.test.string AS redttt ON
    redttt._key = myttt.test_name
    INNER JOIN kafka_tests.analyze.action AS kafkataa
    ON kafkataa.name = myttt.test_name
    WHERE myttt.test_name = '{$name}' LIMIT 1";

    View full-size slide

  66. Apache Kafkaͷ࢝Ίํ
    • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ
    • Confluent͔Βμ΢ϯϩʔυ
    • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ

    Confluentར༻͕Φεεϝ
    • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢

    View full-size slide

  67. Apache Kafka GUI
    • Cluster؅ཧ 

    https://github.com/yahoo/kafka-manager
    • Message؅ཧ 

    https://github.com/landoop/kafka-topics-ui

    https://github.com/ldaniels528/trifecta

    https://github.com/Landoop/kafka-topics-ui

    https://github.com/Landoop/kafka-connect-ui

    ͳͲ

    View full-size slide

  68. ·ͱΊ
    • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ
    ͷ໰୊ղܾͷҰͭ
    • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ
    • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑
    • ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ

    View full-size slide