デブサミ2018 Apache Kafkaの話

by yuuki takezawa

Slide 1

Slide 1 text

Apache KafkaʹΑΔ εέʔϥϒϧ ΞϓϦέʔγϣϯ։ൃ yuuki takezawa   Developers Summit 2018 - 2/16

Slide 2

Slide 2 text

Agenda • What Apache Kafka • Kafka Connect / Kafka Streams • Kappa Architecture • ΞϓϦέʔγϣϯͰ׆༻

Slide 3

Slide 3 text

Scalability • ن໛ಁաੑ  ෛՙͷߴ௿ʹ߹ΘͤͯϦιʔεɾϓʔϧΛ  ֦େɾॖখͰ͖Δ͜ͱ • Ґஔಁաੑ  Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ • ҟछಁաੑ  γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜ ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ

Slide 4

Slide 4 text

What is Apache Kafka?

Slide 5

Slide 5 text

Message Broker

Slide 6

Slide 6 text

Apache Kafka • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ • ϏοάσʔλରԠ • ϑΝΠϧγεςϜར༻Ͱɺ  γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ  (Amazon kinesisͱ΄΅ಉ͡)

Slide 7

Slide 7 text

Apache Kafka + ZooKeeper Architecture

Slide 8

Slide 8 text

Apache Kafka֓ཁ • Producer   ϝοηʔδ഑৴Λߦ͏  ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻ • Consumer   ϝοηʔδߪಡΛߦ͏  ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ  Ұఆظؒอ؅͞ΕΔ • Broker  KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ

Slide 9

Slide 9 text

Apache Kafka֓ཁ • Topic   Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ  ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ • Partition  ෛՙ෼ࢄ༻్ʹར༻  ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ  ͦΕͧΕ͕ॲཧΛߦ͏  ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏

Slide 10

Slide 10 text

Example Partition

Slide 11

Slide 11 text

Producer αϯϓϧ https://github.com/istyle-inc/go- example-developers-summit

Slide 12

Slide 12 text

import ( "github.com/confluentinc/confluent-kafka-go/kafka" ) func (kc *KafkaClient) PrepareProducer() { p, err := kafka.NewProducer(&kafka.ConfigMap{ "bootstrap.servers": *kc.BootstrapServers, "broker.address.family": "v4", "queue.buffering.max.messages": "1000", "client.id": "testingClient", }) // লུ } ઀ଓ͢Δ,BGLBΛࢦఆ

Slide 13

Slide 13 text

Kafka Connect • Kafka Connectͱ͸ɺ  पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ  σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ  ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ  ͕ߦ͑Δ • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ  (java, Scala)

Slide 14

Slide 14 text

Example Kafka Connect &WFOU4PVSDJOH 4UBUF4PVSDJOH ଞΞϓϦέʔγϣϯ  $POTVNFS ૹ৴಺༰Λͦͷ··  &MBTUJDTFBSDI΁

Slide 15

Slide 15 text

Kafka Connect Sink - Elasticsearch

Slide 16

Slide 16 text

bootstrap.servers=192.168.10.10:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 rest.port=8093 plugin.path=/usr/share/java/

Slide 17

Slide 17 text

name=elasticsearch-sink connector.class=io.conﬂuent.connect.elasticsearch.ElasticsearchSinkConnector tasks.max=1 topics=sample.append_employees key.ignore=true connection.url=http://localhost:9200 schema.ignore=true type.name=kafka-connect FMBTUJDTFBSDIDPOOFDU ର৅ͷUPQJDΛબ୒

Slide 18

Slide 18 text

$ conﬂuent load elasticsearch-sink -d \ /etc/kafka-connect-elasticsearch/sample-es.properties $ connect-standalone \ -daemon /etc/schema-registry/elasticsearch-connect.properties \ /etc/kafka-connect-elasticsearch/sample-es.properties DPOOFDUΛొ࿥ సૹ͕։࢝͞Ε·͢

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Command Query Responsibility Segregation

Slide 21

Slide 21 text

Kafka Streams • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ  Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ • ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ  ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ

Slide 22

Slide 22 text

What is Stream?

Slide 23

Slide 23 text

ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ  ετϦʔϜσʔλॲཧͷ໨త • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ

Slide 24

Slide 24 text

Kafka Streams • KStream / KTable • KStream͸ετϦʔϜͰྲྀΕͯ͘Δσʔλ͕KVͰͦͷ ··ྲྀΕͯ͘Δ • KStream͸γϯϓϧʹﬁlterॲཧ΍ɺஔ͖׵͑༻్  ϩά΍ɺπΠʔτɺλΠϜϥΠϯతͳ΋ͷʹ

Slide 25

Slide 25 text

؆୯ͳαϯϓϧ https://github.com/istyle-inc/ developers-summit-kafka-streams

Slide 26

Slide 26 text

def main(args: Array[String]) { println("Kafka Streams Sample.") val config: Properties = { val prop = new Properties() prop.load(new java.io.FileInputStream(args(0).toString)) prop.put(StreamsConfig.APPLICATION_ID_CONFIG, prop.getProperty("sample.app_id")) prop.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, prop.getProperty("sample.bootstrap.servers")) prop.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass) prop.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass) // exactly once prop.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE) prop } } LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG αʔϏεશମͰಉ͡BQQ@JE͸࢖Θͳ͍ ઀ଓ͢ΔTFSWFSΛࢦఆ

Slide 27

Slide 27 text

val ft: String = config.getProperty("sample.stream.topic") val tt: String = config.getProperty("sample.streamed.topic") println("stream topic: from " + ft) println("to " + tt) val stringSerde: Serde[String] = Serdes.String() val builder: StreamsBuilder = new StreamsBuilder val rawStream: KStream[String, String] = builder.stream(ft) val mapStream: KStream[String, String] = rawStream.mapValues(new ValueMapper[String, String] { override def apply(value: String): String = new ElementAppender(value).append() }) mapStream.to(stringSerde, stringSerde, tt) val streams: KafkaStreams = new KafkaStreams(builder.build(), config) streams.start() 4USFBNॲཧର৅UPQJD 4USFBNॲཧޙʹ֨ೲ͢ΔUPQD ,4USFBNͰॲཧ σʔλͷஔ͖׵͑ ॲཧޙʹUPQJD֨ೲΛߦ͏ 4USFBNॲཧ໋ྩܧଓతʹ࣮ߦ͞ΕΔ

Slide 28

Slide 28 text

ϝοηʔδͷܽଛ ૹ৴ࣦഊ BDLड৴ࣦഊ #SPLFSμ΢ϯ ϝοηʔδड৴ࣦഊ

Slide 29

Slide 29 text

Exactly Once • 0.11͔Β௥Ճ • Producer, BrokerؒͷτϥϯβΫγϣϯ  Broker, ConsumerؒͷτϥϯβΫγϣϯ • ਖ਼֬ʹҰ౓͚ͩૹ৴

Slide 30

Slide 30 text

ϝοηʔδ఻ୡͷอূ • At least once semantics  ॏෳΛڐՄ • At most once semantics  ܽଛΛڐՄ • Exactly once semantics  ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍    "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Conﬂuent APACHE KAFKA.   https://www.conﬂuent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর

Slide 31

Slide 31 text

Apache KafkaͰ՝୊ղܾ • ྲྀΕͯ͘Δσʔλʹ߹Θͤͯ  ։ൃऀ͕෼ࢄํ๏Λઃܭ͢Δ͜ͱ͕Ͱ͖Δ • ϩάͷτϥϯβΫγϣϯ  ESɾCQRSͰΞϓϦέʔγϣϯʹऔΓࠐΉ͜ͱ͕Մೳ • Kafka ConnectɺKafka Streams͕ڧྗ  • HadoopΤίγεςϜͰ਌࿨ੑ

Slide 32

Slide 32 text

ڊେԽ͢ΔΞϓϦέʔγϣϯʹ

Slide 33

Slide 33 text

γϯϓϧͳΞϓϦέʔγϣϯߏ੒

Slide 34

Slide 34 text

γϯϓϧͳΞϓϦέʔγϣϯߏ੒

Slide 35

Slide 35 text

ڊେͳΞϓϦέʔγϣϯ΁

Slide 36

Slide 36 text

ෳࡶʹͳΔΞϓϦέʔγϣϯ • σʔλಉظͷෳࡶԽ  ϦΞϧλΠϜͰσʔλ͕ཉ͍͠  ΠϯσοΫεߋ৽͕ఆظ࣮ߦ • ͦͷଞ  Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ • ෳࡶԽ͢Δ࣮૷ίʔυ

Slide 37

Slide 37 text

എܠ • େ͖͘ෳࡶԽ͢ΔΞϓϦέʔγϣϯͷ੹຿  -> ݶΒΕͨػೳͷΈఏڙ͢Δ   -> ϚΠΫϩαʔϏεΑΓʹ͢Δඞཁ͕͋ͬͨ • ೗Կʹεέʔϧ͍͔ͤͯ͘͞  -> ͍͍ϋʔυ΢ΣΞͰؤுΔʹ΋ݶք͕͋Δ • ߴՄ༻ɾো֐଱ੑ  -> σʔλଛࣦ୲อɺϦτϥΠͳͲ͕Մೳͳ΋ͷ͕ʂ

Slide 38

Slide 38 text

௨஌ػೳʹಋೖ • ͳʹͳʹ͞Μ͕Ͳ͜ͰԿ͠·ͨ͠  ԿϙΠϯτ֫ಘ͠·ͨ͠ʂ etc • αʔϏεԣஅͰ࢖ΘΕΔ • ਺100ສ෼ͷEvent (ଟ͍ͱ͖) • σʔλՃ޻ʹRDBMSͱ૊Έ߹ΘͤΔඞཁ͕͋Γɺ  αʔϏεʹΑͬͯ͸ෳࡶʹͳΔ΋ͷ΋͋ΔͨΊɺ  Consumerͷෛՙ෼ࢄΛ༰қʹ͍ͨ͠ • ϝοηʔδଛࣦΛͳΔ΂͘๷͍͗ͨ

Slide 39

Slide 39 text

௨஌ػೳ 1SPEVDFSந৅Խ ϥΠϒϥϦఏڙ ௨஌ొ࿥"1*ఏڙ UPQJDʹ߹Θͤͯ 1BSUJUJPO/ ଞͷαʔϏεͰSBCCJUNRΛར༻ ͍ͯ͠ΔͨΊɺSBCCJUNR΁ ,BGLB$POOFDUʹมߋ༧ఆ $BTTBOESBΫϥελ

Slide 40

Slide 40 text

ಋೖޙ • Ұ೥ؒͷ͏ͪ໨ཱͬͨো֐͸ͳ͠  • ϝοηʔδ֬ೝ͕༰қͰɺૄ௨֬ೝͳͲॏๅ  • KafkaɺCassandraͷ૊Έ߹ΘͤͰࡹ͚͗ͯ͢͠·͍ɺ  ͍ͭ΋εΧεΧ  • rabbitmqͷόοΫΞοϓͱͯ͠΋Քಇ  *ࣾ಺Ͱrabbitmqར༻αʔϏεͷํ͕ଟ͍ͨΊ

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Kafka Connect Source - Rabbitmq

Slide 43

Slide 43 text

name=named-kafka-connect-rabbitmq tasks.max=1 connector.class=com.github.jcustenborder.kafka.connect.rabbitmq.RabbitMQSour ceConnector rabbitmq.prefetch.count=500 rabbitmq.automatic.recovery.enabled=true rabbitmq.network.recovery.interval.ms=10000 rabbitmq.topology.recovery.enabled=true rabbitmq.queue=action_api.create_object kafka.topic=rabbitmq.source_topic rabbitmq.host=rabbitmqhost.local rabbitmq.password=password rabbitmq.username=username

Slide 44

Slide 44 text

$ conﬂuent load named-kafka-connect-rabbitmq -d \ /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties $ connect-standalone \ -daemon /etc/schema-registry/connect-json-standalone.properties \ /etc/kafka-connect-rabbitmq/rabbitmq-source-connect.properties

Slide 45

Slide 45 text

RabbitMQ -> Kafka ,BGLB$POOFDUͰ ٵ্͍͛ ,BGLB4USFBNTͰ UPQJDৼΓ෼͚ LBGLBDPOOFDUIEGTͰ IEGTʹग़ྗ

Slide 46

Slide 46 text

Lambda Architecture

Slide 47

Slide 47 text

BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ  σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠ • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠ • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠ • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔

Slide 48

Slide 48 text

BigData΁ͷΞϓϩʔν • σʔλͦͷ΋ͷͷू໿ • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ  ߴ଎ͳMessageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ

Slide 49

Slide 49 text

Lambda Architecture • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒ • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳ ͲΛ୲౰͢Δ -> Hadoop(MapReduce), Spark • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ  Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚  Spark, Storm, Kafka, Cassandra etc.. • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫  αʔϏεʹΑͬͯ͸೉қ౓͕ߴ͍ɾɾʂ

Slide 50

Slide 50 text

Lambda Architecture

Slide 51

Slide 51 text

Kappa Architecture

Slide 52

Slide 52 text

KappaΞʔΩςΫνϟ

Slide 53

Slide 53 text

KappaΞʔΩςΫνϟ

Slide 54

Slide 54 text

def main(args: Array[String]) { val kafkaParams = Map[String, Object]( "bootstrap.servers" -> "localhost:9092", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "kafka_builderscon_stream", "auto.offset.reset" -> "latest", "enable.auto.commit" -> (false: java.lang.Boolean) ) // val spark = SparkSession .builder .master("local[*]") .appName("buildersconSmaple") .conﬁg("spark.cassandra.connection.host", "192.168.10.10") .getOrCreate() val streamingContext = new StreamingContext(spark.sparkContext, Seconds(5)) LBGLB4USFBNͰ࣮ߦ͞ΕΔΔEFG ,BGLB઀ଓ৘ใ ࠷৽ͷྲྀΕͯདྷͨσʔλΛ࢖ͬͯৗʹॲཧ 4QBSL"QQ໊ 4USFBNJOHॲཧ݁ՌΛ$BTTBOESBʹ ඵ͓͖ʹશσʔλΛॻ͖ࠐΉ

Slide 55

Slide 55 text

streamingContext.checkpoint("/tmp/") val topics = Array("message-topic") val stream = KafkaUtils.createDirectStream[String, String]( streamingContext, PreferConsistent, Subscribe[String, String](topics, kafkaParams) ) val pairs = stream.map(record => (record.value, 1)) val count = pairs.updateStateByKey(updateFunc) ॲཧσʔλͷܦաΛॻ͖ࠐΉ ࢮΜͰ΋͔͜͜Β΍Γ௚͢ ,BGLB5PQJDͷσʔλΛ 4QBSLͷ4USFBNʹྲྀ͠ࠐΉ

Slide 56

Slide 56 text

count.foreachRDD((rdd, time) => { val count = rdd.count() if (count > 0) { rdd.map(record => ("spark", streamMessageParse(record._1.toString).message, record._2)) .saveToCassandra("builderscon", "counter", SomeColumns("stream", "message", "counter")) } }) count.print() streamingContext.start() streamingContext.awaitTermination() } ྲྀΕͯ͘Δ3%%ͷॲཧ։࢝ ूܭ݁ՌΛ$BTTBOESBͷΧϥϜʹNBQ ͯ͠ॻ͖ࠐΉ

Slide 57

Slide 57 text

for Application

Slide 58

Slide 58 text

ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • ProducerͰόϦσʔγϣϯ  -> ࠷௿ݶͷܕकΔ  -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍  • ConsumerͰόϦσʔγϣϯ  ड৴ͨ͠ϝοηʔδΛௐ΂ɺ  ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ

Slide 59

Slide 59 text

ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • Kafka Streamͱ͍͏બ୒ࢶ  java·ͨ͸ScalaͰ࣮૷Մೳ  -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍  • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏  -> ϚΠΫϩαʔϏεͰ͋Ε͹

Slide 60

Slide 60 text

PipelineDBͱ͍͏બ୒ࢶ • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ  ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠  • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍  • PostgreSQL Compatible  • PDOͰ઀ଓՄೳʂ

Slide 61

Slide 61 text

PipelineDBͱ͍͏બ୒ࢶ CREATE EXTENSION pipeline_kafka; SELECT pipeline_kafka.add_broker(‘localhost:9092'); CREATE STREAM logs_stream (payload json); CREATE CONTINUOUS VIEW message_count AS SELECT COUNT(*) FROM logs_stream;

Slide 62

Slide 62 text

KSQL • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ  SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ • ݱࡏDeveloper Preview • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢

Slide 63

Slide 63 text

ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ • PrestoΛར༻͠ɺ  RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ  RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ  KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλֻ͚߹Θͤ

Slide 64

Slide 64 text

for Prestodb • ܕఆٛ etc/kafka/table໊.json • ઀ଓ৘ใ “etc/catalog/kafka_tests.properties” • prestoͷΧλϩάʹ઀ଓ৘ใΛهड़ • topic໊͸ hoge.fuga ܗࣜͱ͢Δ

Slide 65

Slide 65 text

{ "tableName": “action", "schemaName": "analyze", "topicName": "analyze.action", "message": { "dataFormat": "json", "ﬁelds": [ { "name": "uuid", "mapping": "uuid", "type": "VARCHAR" }, { "name": "uri", "mapping": "uri", "type": "VARCHAR" }, { "name": "name", "mapping": "name", "type": "VARCHAR" } ] } LBGLBͷUPQJD৘ใΛهࡌ ΧϥϜΛఆٛϓϦϛςΟϒͳܕʹม׵

Slide 66

Slide 66 text

connector.name=kafka kafka.nodes=127.0.0.1:9092 kafka.table-names=analyze.action kafka.hide-internal-columns=false DPOODUPS໊͸LBGLB LBGLBͷΫϥελʔΛࢦఆ LBGLBͷUPQJDΛࢦఆ LBGLBͷγεςϜΧϥϜ༗ແ

Slide 67

Slide 67 text

SELECT redttt._key, redttt._value, test_id, test_name, created_at, uri, uuid FROM my_tests.testing.tests AS myttt INNER JOIN red_tests.test.string AS redttt ON redttt._key = myttt.test_name INNER JOIN kafka_tests.analyze.action AS kafkataa ON kafkataa.name = myttt.test_name WHERE myttt.test_name = '{$name}' LIMIT 1";

Slide 68

Slide 68 text

Apache Kafkaͷ࢝Ίํ • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ • Conﬂuent͔Βμ΢ϯϩʔυ • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ  Conﬂuentར༻͕Φεεϝ • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢

Slide 69

Slide 69 text

Apache Kafka GUI • Cluster؅ཧ   https://github.com/yahoo/kafka-manager • Message؅ཧ   https://github.com/landoop/kafka-topics-ui  https://github.com/ldaniels528/trifecta  https://github.com/Landoop/kafka-topics-ui  https://github.com/Landoop/kafka-connect-ui  ͳͲ

Slide 70

Slide 70 text

·ͱΊ • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ ͷ໰୊ղܾͷҰͭ • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑ • ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ