Upgrade to Pro — share decks privately, control downloads, hide ads and more …

デブサミ2018 Apache Kafkaの話

17d4ef53b432ebf7c566fd6a11345570?s=47 yuuki takezawa
February 02, 2018

デブサミ2018 Apache Kafkaの話

Developers Summit2018でのスライド資料です
"Apache Kafkaによるスケーラブル アプリケーション開発"

17d4ef53b432ebf7c566fd6a11345570?s=128

yuuki takezawa

February 02, 2018
Tweet

Transcript

  1. Apache KafkaʹΑΔ εέʔϥϒϧ ΞϓϦέʔγϣϯ։ൃ <V2> yuuki takezawa <ytake>
 Developers Summit

    2018 - 2/16
  2. Agenda • What Apache Kafka • Kafka Connect / Kafka

    Streams • Kappa Architecture • ΞϓϦέʔγϣϯͰ׆༻
  3. Scalability • ن໛ಁաੑ
 ෛՙͷߴ௿ʹ߹ΘͤͯϦιʔεɾϓʔϧΛ
 ֦େɾॖখͰ͖Δ͜ͱ • Ґஔಁաੑ
 Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ •

    ҟछಁաੑ
 γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜ ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ
  4. What is Apache Kafka?

  5. Message Broker

  6. Apache Kafka • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ • ϏοάσʔλରԠ • ϑΝΠϧγεςϜར༻Ͱɺ


    γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ
 (Amazon kinesisͱ΄΅ಉ͡)
  7. Apache Kafka + ZooKeeper Architecture

  8. Apache Kafka֓ཁ • Producer 
 ϝοηʔδ഑৴Λߦ͏
 ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻ • Consumer 


    ϝοηʔδߪಡΛߦ͏
 ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ
 Ұఆظؒอ؅͞ΕΔ • Broker
 KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
  9. Apache Kafka֓ཁ • Topic 
 Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ
 ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ • Partition
 ෛՙ෼ࢄ༻్ʹར༻


    ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ
 ͦΕͧΕ͕ॲཧΛߦ͏
 ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
  10. Example Partition

  11. Producer αϯϓϧ https://github.com/istyle-inc/go- example-developers-summit

  12. import ( "github.com/confluentinc/confluent-kafka-go/kafka" ) func (kc *KafkaClient) PrepareProducer() { p,

    err := kafka.NewProducer(&kafka.ConfigMap{ "bootstrap.servers": *kc.BootstrapServers, "broker.address.family": "v4", "queue.buffering.max.messages": "1000", "client.id": "testingClient", }) // লུ } ઀ଓ͢Δ,BGLBΛࢦఆ
  13. Kafka Connect • Kafka Connectͱ͸ɺ
 पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ
 σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ


    ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ
 ͕ߦ͑Δ • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ
 (java, Scala)
  14. Example Kafka Connect &WFOU4PVSDJOH 4UBUF4PVSDJOH ଞΞϓϦέʔγϣϯ
 $POTVNFS ૹ৴಺༰Λͦͷ··
 &MBTUJDTFBSDI΁

  15. Kafka Connect Sink - Elasticsearch

  16. bootstrap.servers=192.168.10.10:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false offset.storage.file.filename=/tmp/connect.offsets

    offset.flush.interval.ms=10000 rest.port=8093 plugin.path=/usr/share/java/
  17. name=elasticsearch-sink connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector tasks.max=1 topics=sample.append_employees key.ignore=true connection.url=http://localhost:9200 schema.ignore=true type.name=kafka-connect FMBTUJDTFBSDIDPOOFDU ର৅ͷUPQJDΛબ୒

  18. $ confluent load elasticsearch-sink -d \ /etc/kafka-connect-elasticsearch/sample-es.properties $ connect-standalone \

    -daemon /etc/schema-registry/elasticsearch-connect.properties \ /etc/kafka-connect-elasticsearch/sample-es.properties DPOOFDUΛొ࿥ సૹ͕։࢝͞Ε·͢
  19. None
  20. Command Query Responsibility Segregation

  21. Kafka Streams • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ
 Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ •

    ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ
 ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ
  22. What is Stream?

  23. ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ
 ετϦʔϜσʔλॲཧͷ໨త • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ

    • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
  24. Kafka Streams • KStream / KTable • KStream͸ετϦʔϜͰྲྀΕͯ͘Δσʔλ͕KVͰͦͷ ··ྲྀΕͯ͘Δ •

    KStream͸γϯϓϧʹfilterॲཧ΍ɺஔ͖׵͑༻్
 ϩά΍ɺπΠʔτɺλΠϜϥΠϯతͳ΋ͷʹ
  25. ؆୯ͳαϯϓϧ https://github.com/istyle-inc/ developers-summit-kafka-streams

  26. def main(args: Array[String]) { println("Kafka Streams Sample.") val config: Properties

    = { val prop = new Properties() prop.load(new java.io.FileInputStream(args(0).toString)) prop.put(StreamsConfig.APPLICATION_ID_CONFIG, prop.getProperty("sample.app_id")) prop.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, prop.getProperty("sample.bootstrap.servers")) prop.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass) prop.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass) // exactly once prop.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE) prop } } LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG αʔϏεશମͰಉ͡BQQ@JE͸࢖Θͳ͍ ઀ଓ͢ΔTFSWFSΛࢦఆ
  27. val ft: String = config.getProperty("sample.stream.topic") val tt: String = config.getProperty("sample.streamed.topic")

    println("stream topic: from " + ft) println("to " + tt) val stringSerde: Serde[String] = Serdes.String() val builder: StreamsBuilder = new StreamsBuilder val rawStream: KStream[String, String] = builder.stream(ft) val mapStream: KStream[String, String] = rawStream.mapValues(new ValueMapper[String, String] { override def apply(value: String): String = new ElementAppender(value).append() }) mapStream.to(stringSerde, stringSerde, tt) val streams: KafkaStreams = new KafkaStreams(builder.build(), config) streams.start() 4USFBNॲཧର৅UPQJD 4USFBNॲཧޙʹ֨ೲ͢ΔUPQD ,4USFBNͰॲཧ σʔλͷஔ͖׵͑ ॲཧޙʹUPQJD֨ೲΛߦ͏ 4USFBNॲཧ໋ྩܧଓతʹ࣮ߦ͞ΕΔ
  28. ϝοηʔδͷܽଛ ૹ৴ࣦഊ BDLड৴ࣦഊ #SPLFSμ΢ϯ ϝοηʔδड৴ࣦഊ

  29. Exactly Once • 0.11͔Β௥Ճ • Producer, BrokerؒͷτϥϯβΫγϣϯ
 Broker, ConsumerؒͷτϥϯβΫγϣϯ •

    ਖ਼֬ʹҰ౓͚ͩૹ৴
  30. ϝοηʔδ఻ୡͷอূ • At least once semantics
 ॏෳΛڐՄ • At most

    once semantics
 ܽଛΛڐՄ • Exactly once semantics
 ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍
 
 "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 
 https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
  31. Apache KafkaͰ՝୊ղܾ • ྲྀΕͯ͘Δσʔλʹ߹Θͤͯ
 ։ൃऀ͕෼ࢄํ๏Λઃܭ͢Δ͜ͱ͕Ͱ͖Δ • ϩάͷτϥϯβΫγϣϯ
 ESɾCQRSͰΞϓϦέʔγϣϯʹऔΓࠐΉ͜ͱ͕Մೳ • Kafka

    ConnectɺKafka Streams͕ڧྗ
 • HadoopΤίγεςϜͰ਌࿨ੑ
  32. ڊେԽ͢ΔΞϓϦέʔγϣϯʹ

  33. γϯϓϧͳΞϓϦέʔγϣϯߏ੒

  34. γϯϓϧͳΞϓϦέʔγϣϯߏ੒

  35. ڊେͳΞϓϦέʔγϣϯ΁

  36. ෳࡶʹͳΔΞϓϦέʔγϣϯ • σʔλಉظͷෳࡶԽ
 ϦΞϧλΠϜͰσʔλ͕ཉ͍͠
 ΠϯσοΫεߋ৽͕ఆظ࣮ߦ • ͦͷଞ
 Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ • ෳࡶԽ͢Δ࣮૷ίʔυ

  37. എܠ • େ͖͘ෳࡶԽ͢ΔΞϓϦέʔγϣϯͷ੹຿
 -> ݶΒΕͨػೳͷΈఏڙ͢Δ 
 -> ϚΠΫϩαʔϏεΑΓʹ͢Δඞཁ͕͋ͬͨ • ೗Կʹεέʔϧ͍͔ͤͯ͘͞


    -> ͍͍ϋʔυ΢ΣΞͰؤுΔʹ΋ݶք͕͋Δ • ߴՄ༻ɾো֐଱ੑ
 -> σʔλଛࣦ୲อɺϦτϥΠͳͲ͕Մೳͳ΋ͷ͕ʂ
  38. ௨஌ػೳʹಋೖ • ͳʹͳʹ͞Μ͕Ͳ͜ͰԿ͠·ͨ͠
 ԿϙΠϯτ֫ಘ͠·ͨ͠ʂ etc • αʔϏεԣஅͰ࢖ΘΕΔ • ਺100ສ෼ͷEvent (ଟ͍ͱ͖)

    • σʔλՃ޻ʹRDBMSͱ૊Έ߹ΘͤΔඞཁ͕͋Γɺ
 αʔϏεʹΑͬͯ͸ෳࡶʹͳΔ΋ͷ΋͋ΔͨΊɺ
 Consumerͷෛՙ෼ࢄΛ༰қʹ͍ͨ͠ • ϝοηʔδଛࣦΛͳΔ΂͘๷͍͗ͨ
  39. ௨஌ػೳ 1SPEVDFSந৅Խ ϥΠϒϥϦఏڙ ௨஌ొ࿥"1*ఏڙ UPQJDʹ߹Θͤͯ 1BSUJUJPO/ ଞͷαʔϏεͰSBCCJUNRΛར༻ ͍ͯ͠ΔͨΊɺSBCCJUNR΁ ,BGLB$POOFDUʹมߋ༧ఆ $BTTBOESBΫϥελ

  40. ಋೖޙ • Ұ೥ؒͷ͏ͪ໨ཱͬͨো֐͸ͳ͠
 • ϝοηʔδ֬ೝ͕༰қͰɺૄ௨֬ೝͳͲॏๅ
 • KafkaɺCassandraͷ૊Έ߹ΘͤͰࡹ͚͗ͯ͢͠·͍ɺ
 ͍ͭ΋εΧεΧ
 • rabbitmqͷόοΫΞοϓͱͯ͠΋Քಇ


    *ࣾ಺Ͱrabbitmqར༻αʔϏεͷํ͕ଟ͍ͨΊ
  41. None
  42. Kafka Connect Source - Rabbitmq

  43. name=named-kafka-connect-rabbitmq tasks.max=1 connector.class=com.github.jcustenborder.kafka.connect.rabbitmq.RabbitMQSour ceConnector rabbitmq.prefetch.count=500 rabbitmq.automatic.recovery.enabled=true rabbitmq.network.recovery.interval.ms=10000 rabbitmq.topology.recovery.enabled=true rabbitmq.queue=action_api.create_object kafka.topic=rabbitmq.source_topic

    rabbitmq.host=rabbitmqhost.local rabbitmq.password=password rabbitmq.username=username
  44. $ confluent load named-kafka-connect-rabbitmq -d \ /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties $ connect-standalone \

    -daemon /etc/schema-registry/connect-json-standalone.properties \ /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties
  45. RabbitMQ -> Kafka ,BGLB$POOFDUͰ ٵ্͍͛ ,BGLB4USFBNTͰ UPQJDৼΓ෼͚ LBGLBDPOOFDUIEGTͰ IEGTʹग़ྗ

  46. Lambda Architecture

  47. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ
 σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠ • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠ •

    ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠ • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔
  48. BigData΁ͷΞϓϩʔν • σʔλͦͷ΋ͷͷू໿ • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ
 ߴ଎ͳMessageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ

  49. Lambda Architecture • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒ • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳ ͲΛ୲౰͢Δ -> Hadoop(MapReduce), Spark

    • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ
 Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚
 Spark, Storm, Kafka, Cassandra etc.. • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫
 αʔϏεʹΑͬͯ͸೉қ౓͕ߴ͍ɾɾʂ
  50. Lambda Architecture

  51. Kappa Architecture

  52. KappaΞʔΩςΫνϟ

  53. KappaΞʔΩςΫνϟ

  54. def main(args: Array[String]) { val kafkaParams = Map[String, Object]( "bootstrap.servers"

    -> "localhost:9092", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "kafka_builderscon_stream", "auto.offset.reset" -> "latest", "enable.auto.commit" -> (false: java.lang.Boolean) ) // val spark = SparkSession .builder .master("local[*]") .appName("buildersconSmaple") .config("spark.cassandra.connection.host", "192.168.10.10") .getOrCreate() val streamingContext = new StreamingContext(spark.sparkContext, Seconds(5)) LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG ,BGLB઀ଓ৘ใ ࠷৽ͷྲྀΕͯདྷͨσʔλΛ࢖ͬͯৗʹॲཧ 4QBSL"QQ໊ 4USFBNJOHॲཧ݁ՌΛ$BTTBOESBʹ ඵ͓͖ʹશσʔλΛॻ͖ࠐΉ
  55. streamingContext.checkpoint("/tmp/") val topics = Array("message-topic") val stream = KafkaUtils.createDirectStream[String, String](

    streamingContext, PreferConsistent, Subscribe[String, String](topics, kafkaParams) ) val pairs = stream.map(record => (record.value, 1)) val count = pairs.updateStateByKey(updateFunc) ॲཧσʔλͷܦաΛॻ͖ࠐΉ ࢮΜͰ΋͔͜͜Β΍Γ௚͢ ,BGLB5PQJDͷσʔλΛ 4QBSLͷ4USFBNʹྲྀ͠ࠐΉ
  56. count.foreachRDD((rdd, time) => { val count = rdd.count() if (count

    > 0) { rdd.map(record => ("spark", streamMessageParse(record._1.toString).message, record._2)) .saveToCassandra("builderscon", "counter", SomeColumns("stream", "message", "counter")) } }) count.print() streamingContext.start() streamingContext.awaitTermination() } ྲྀΕͯ͘Δ3%%ͷॲཧ։࢝ ूܭ݁ՌΛ$BTTBOESBͷΧϥϜʹNBQ ͯ͠ॻ͖ࠐΉ
  57. for Application

  58. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • ProducerͰόϦσʔγϣϯ
 -> ࠷௿ݶͷܕकΔ
 -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍
 • ConsumerͰόϦσʔγϣϯ
 ड৴ͨ͠ϝοηʔδΛௐ΂ɺ


    ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ
  59. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • Kafka Streamͱ͍͏બ୒ࢶ
 java·ͨ͸ScalaͰ࣮૷Մೳ
 -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍
 • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏
 ->

    ϚΠΫϩαʔϏεͰ͋Ε͹
  60. PipelineDBͱ͍͏બ୒ࢶ • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ
 ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠
 • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍
 • PostgreSQL Compatible


    • PDOͰ઀ଓՄೳʂ
  61. PipelineDBͱ͍͏બ୒ࢶ CREATE EXTENSION pipeline_kafka; SELECT pipeline_kafka.add_broker(‘localhost:9092'); CREATE STREAM logs_stream (payload

    json); CREATE CONTINUOUS VIEW message_count AS SELECT COUNT(*) FROM logs_stream;
  62. KSQL • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ
 SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ • ݱࡏDeveloper Preview • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢

  63. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ • PrestoΛར༻͠ɺ
 RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ
 RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ
 KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλֻ͚߹Θͤ

  64. for Prestodb • ܕఆٛ etc/kafka/table໊.json • ઀ଓ৘ใ “etc/catalog/kafka_tests.properties” • prestoͷΧλϩάʹ઀ଓ৘ใΛهड़

    • topic໊͸ hoge.fuga ܗࣜͱ͢Δ
  65. { "tableName": “action", "schemaName": "analyze", "topicName": "analyze.action", "message": { "dataFormat":

    "json", "fields": [ { "name": "uuid", "mapping": "uuid", "type": "VARCHAR" }, { "name": "uri", "mapping": "uri", "type": "VARCHAR" }, { "name": "name", "mapping": "name", "type": "VARCHAR" } ] } LBGLBͷUPQJD৘ใΛهࡌ ΧϥϜΛఆٛϓϦϛςΟϒͳܕʹม׵
  66. connector.name=kafka kafka.nodes=127.0.0.1:9092 kafka.table-names=analyze.action kafka.hide-internal-columns=false DPOODUPS໊͸LBGLB LBGLBͷΫϥελʔΛࢦఆ LBGLBͷUPQJDΛࢦఆ LBGLBͷγεςϜΧϥϜ༗ແ

  67. SELECT redttt._key, redttt._value, test_id, test_name, created_at, uri, uuid FROM my_tests.testing.tests

    AS myttt INNER JOIN red_tests.test.string AS redttt ON redttt._key = myttt.test_name INNER JOIN kafka_tests.analyze.action AS kafkataa ON kafkataa.name = myttt.test_name WHERE myttt.test_name = '{$name}' LIMIT 1";
  68. Apache Kafkaͷ࢝Ίํ • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ • Confluent͔Βμ΢ϯϩʔυ • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ


    Confluentར༻͕Φεεϝ • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢
  69. Apache Kafka GUI • Cluster؅ཧ 
 https://github.com/yahoo/kafka-manager • Message؅ཧ 


    https://github.com/landoop/kafka-topics-ui
 https://github.com/ldaniels528/trifecta
 https://github.com/Landoop/kafka-topics-ui
 https://github.com/Landoop/kafka-connect-ui
 ͳͲ
  70. ·ͱΊ • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ ͷ໰୊ղܾͷҰͭ • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑ •

    ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ