Apache Kafka & Kafka Connectを使ったデータ連携パターン(改めETLの実装)

Transcript

1. Apache Kafka & Kafka ConnectΛ ʹ࢖ͬͨσʔλ࿈ܞύλʔϯ(վΊETLͷ࣮૷) Future Architect ,Inc Keigo Suda 2016/11/25 D&S Data Night vol.04
2. 12 ! Kafka ConnectΛETLπʔϧͱͯ͠ར༻͢Δػձ͕͋ͬͨͷͰͦͷࡍʹ ܦݧͨ͠ҎԼʹ͍ͭͯڞ༗(Ұ௨Γ͸·Γ౗ͨ͠) ࢖͍Ͳ͜Ζ (؆୯ʹ)࣮૷ϙΠϯτ/ϋϚΓͲ͜Ζ ΋͖ͯ͘
3. 13 ࣗݾ঺հ ! Future Architect ,Inc ਢా ܡޔ (ͩ͢ ͚͍͝) ! ΤϯλʔϓϥΠζͳͱ͜ΖͰେ͖͍σʔλΛѻ͏࢓ࣄΛͯ͠·͢ ! ࠷ۙIoT·ΘΓͷσʔλج൫Λ࡞ͬͯ·͢
4. 14 ຊ୊ʹ͸͍Δલʹ
5. 15 Kafka Connect͝ଘ஌Ͱ͔͢ʁ
6. 16 ! Kafka ver 0.9͔Βಋೖ͞Εͨ৽ػೳ ! Kafkaͱͷ࿈ܞϑϨʔϜϫʔΫ ! ϓϥΨϒϧͳػ ! ઃఆϑΝΠϧʹΑΔૢ࡞ ! σʔλΛͲ͏ൈ͔͘/ೖΕΔ͔ͷΈ࣮૷ Kafka Connect ͜͜ͷ෦෼
7. 17 Connectors https://www.confluent.io/product/connectors/
8. ͲΕͲΕɺKafka ConnectͰݕࡧͬͱ
10. ͳΜ͔ͩ࿐ࠎʹ͋Ε͚ͩͲɺ෺͸ࢼ͠
11. 21 ৘ใ͕΄΅օແ
12. 22 ! ࠃ֎ɾࠃ಺ͱ΋ʹར༻ࣄྫ͸গͳ͍ҹ৅ ! υΩϡϝϯτࣗମ΋ΠϚΠνɺࡉ͔͍֬ೝ͸௚઀ιʔεΛ֬ೝ ! ͍Ζ͍ΖͱϓϥάΠϯ͕ެ։͞Ε͖͍ͯͯΔ͕ɺΠϚΠν࣮૷ํ๏΋౷ Ұ͕ͱΕ͍ͯͳ͍ײ͡Ͱ͓खຊ͕গͳ͍ Kafka Connectͷݱঢ়(ݸਓతҙݟ)
13. 23 ETLʹ͓͚ΔKakfa Connect(ݸਓతҙݟ) P T // / / AC KM I M KM
14. Kafka ConnectͷΩϗϯ
15. 25 http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started Kakfa Connectͷجຊ ͕͜͜Kafka Connect
16. 26 σʔλ࿈ܞݩ σʔλ࿈ܞͷδϣϒ ࣮ࡍͷσʔλ࿈ܞ୯Ґ ࣮ࡍʹσʔλίϐʔΛߦ͏ॲཧ Kakfa Connectͷجຊ http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
17. 27 Stream & Partition(RDBͷྫ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
18. 28 σʔλ࿈ܞݩ ࣮ࡍͷσʔλ࿈ܞ୯Ґ ࣮ࡍʹσʔλίϐʔΛߦ͏ॲཧ N:1 Stream & Partition(RDBͷྫ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
19. 29 σʔλ࿈ܞݩ σʔλ࿈ܞͷδϣϒ ࣮ࡍͷσʔλ࿈ܞ୯Ґ ࣮ࡍʹσʔλίϐʔΛߦ͏ॲཧ Kakfa Connectͷجຊ http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
20. 30 Worker & Connector http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
21. 31 ͋ ͋ ͊ ͜ Ε Ҏ ্ ༻ ޠ Λ ૿ ΍ ͢ ͳ ɾ ɾ ɾ CONNECTER WORKER STREAM PARTITION STANDALONE MODE DISTRIBUTED MODE TASK
22. 32 $ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties ࣮ࡍͷར༻(Standalone mode) ! ίϚϯυϥΠϯ͔ΒσʔϞϯͱͯ͠ىಈ ! ҎԼΛىಈ࣌ʹࢦఆ͢Δඞཁ͋Γ ! ࣮ߦϓϩηεͰ͋ΔWorkerͷઃఆϑΝΠϧ ! ࣮ࡍʹར༻͢ΔConnectorಛ༗ͷઃఆϑΝΠϧ Workerઃఆ Connectorઃఆ
23. 33 ઃఆϑΝΠϧͷத਎ name=local-file-source connector.class=FileStreamSource tasks.max=1 file=test.txt topic=connect-test bootstrap.servers=localhost:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter ... Workerઃఆ Connectorઃఆ
24. ࣮ ૷
25. 35 ! ϓϥάΠϯ΋৭ʑͱެ։͞ΕΔΑ͏ʹͳ͖ͬͯͨ ! ͨͩ͠ɺ·ͩ͜Ε͔Βͳײ͡ͳͷͰࣗ෼Ͱॻ͘ػձ΋ଟ͍ ! ࣮૷ͷओཁͳϙΠϯτͱETL΁ͷੜ͔͠ํʹ͍ͭͯ؆୯ʹ ͔͜͜Βͷ͸ͳ͠
26. 36 ! Kafka Connect͸SourceॲཧͱSinkॲཧ͕͋Γ·͢ɻ ! Sink/Source͸ߟ͑ํ͸ಉ͡Ͱ͕͢ɺSinkͷํ͕࣮૷্গ͠Θ͔Γ΍͢ ͍ͷͰͪ͜ΒΛྫʹ࣮૷ʹ͍ͭͯઆ໌͠·͢ɻ Kafka Connectͷ࣮૷ Source Sink
27. 37 ! ࠷௿ݶɺConnector/TaskΛ࣮૷͢Δ͚ͩͰOK ! ྆ํͱ΋͢ͰʹAPI͕༻ҙ͞Ε͍ͯΔͷͰத਎ΛຒΊΔ͚ͩͰOK σʔλ࿈ܞͷδϣϒ(͜͜) ࣮ࡍʹσʔλίϐʔΛߦ͏ॲཧ(͜͜) ࣮૷Օॴ
28. 38 Connector(΄΅ςϯϓϨ)
29. 39 େࣄͳͷ͸͜͜!!(sourceͷ৔߹͸pull) Task
30. 40 ! ࣮૷ͨ͠TaskͱConnectorΛίϯύΠϧ ! Ϋϥεύε͕௨͍ͬͯΔͱ͜Ζʹ഑ஔ͢Δ͚ͩʂ Deploy name=sample-sink connector.class=SampleSinkConnector tasks.max=1 topic=connect-test ɾɾɾɾ Connectorઃఆ
31. ·ͱΊ
32. 42 ! Kafkaͷ࢓૊ΈΛ͠Βͳͯ͘΋Ͱ͖Δ୯७ͳσʔλίϐʔ΍ϑΟϧλϦ ϯά౳Λ؆୯ʹ࣮૷Ͱ͖Δɻ ! ઃఆϑΝΠϧϕʔεͰͷૢ࡞͕ՄೳͰɺ։ൃऀؒͰͷ࣮૷ͷόϥ͖ͭΛ ͓͑͞ΒΕΔɻ Kafka Connectͷྑ͍ͱ͜
33. 43 ! ෳࡶͳॲཧ͸Ͱ͖ͳ͘ͳ͍ͳ͕ɺͦ΋ͦ΋৘ใগͳ͍͠ɺ࣮૷ྫ΋গͳ ͍ͷͰɺجຊιʔεΛ௥͍ͳ͕Βͷ࣮૷ʢͩͬͨΒී௨ʹAPI࢖ͬͨํ ͕޾ͤʣ Kafka Connectͷѱ͍ͱ͜
34. 44 ࢦఆͨ͠ΠϯΫϦϝϯτϞʔυΛ൑ผ ͋ͱ͸ΫΤϦΛ૊Έཱ͍ͯͯΔ͚ͩ Μʁ͜Μͳແअؾʹେৎ෉ʁ ࠩ෼൓өྫ(kafka-connect-jdbc)
35. 45 ! ศརϓϥάΠϯΛ࢖͍͍ͨ࣌͸ɺ΄ͱΜͲ͕Confluent੡Ͱɺ΋Εͳ͘ Confuluent Platformͷػೳ܈ʹґଘͤ͑͞ΔΛ͑ͳ͍ɻ Kafka Connectͷѱ͍ͱ͜
36. 46 ! KakfaΛհͨ͠୯७ͳσʔλίϐʔʹ͸Ή͍͍ͯΔ(ͱࢥ͏) ! Confluent…. ·ͱΊ