Tweet
Tweet

Slide 1

Slide 1 text

Embulkʹ଍Γͳ͍ 5ͭͷ͜ͱ 2015-12-15 Embulk Meetup Tokyo #2 @Civitaspo

Slide 2

Slide 2 text

͓͜ͱΘΓ

Slide 3

Slide 3 text

࡞ͬͨπʔϧͷ࿩͕ ग़͖ͯ·͕͢

Slide 4

Slide 4 text

ϦϦʔε͠·ͤΜʂ

Slide 5

Slide 5 text

͝ΊΜͳ͍͞(><)

Slide 6

Slide 6 text

ཧ༝ɿ͋Δਓ͔ΒͷҰ੠

Slide 7

Slide 7 text

ࠓ͜ͷπʔϧΛ ϦϦʔε͢Δͱ ੈͷதΛࠞཚͷӔʹ ר͖ࠐΉ͜ͱ ʹͳͬͯ͠·͏ͩΖ͏ɻ

Slide 8

Slide 8 text

ࠞཚͷछʹͳΔͩΖ͏ػೳ͸ ʮEmbulk Projectʹظ଴͢Δ͜ͱʯ ͱ͍͏߲໨Ͱઆ໌͠·͢

Slide 9

Slide 9 text

Ͱ͸࢝Ί·͢ʂ

Slide 10

Slide 10 text

Embulkʹ଍Γͳ͍ 5ͭͷ͜ͱ 2015-12-15 Embulk Meetup Tokyo #2 @Civitaspo

Slide 11

Slide 11 text

ࣗݾ঺հ தࢁوത (@Civitaspo) • DeNA: ೖࣾ3೥໨ • ෼ੳܥΠϯϑϥΤϯδχΞ • σʔλճऩ؀ڥߏங • Hadoopӡ༻…etc. • Perl / Ruby / Java • ৽ଔͰձࣾʹೖͬͯॳΊͯί ϯιʔϧ։͍ͨস

Slide 12

Slide 12 text

࡞ͬͨEmbulk Pluginୡ embulk-input-hdfs embulk-output-hdfs embulk-output-sftp embulk-filter-join-file embulk-filter-json-key embulk-filter-expand-json embulk-filter-flatten-json embulk-filter-distinct ࣭໰΍ཁ๬ɺΞυόΠε౳͋Ε͹twitterͰʂ

Slide 13

Slide 13 text

ͪͳΈʹ શ෦Java PluginͰ͕͢

Slide 14

Slide 14 text

๻͸Intellij࢖ͬͯॻ͍ͯ·͢ vim!! vim!!

Slide 15

Slide 15 text

ΞδΣϯμ • ฐࣾͷBulkdataࣄ৘ • ߏஙͨ͠Embulkج൫ • ଍Γͳ͔͔ͬͨΒิͬͨػೳ̑ͭ • Embulk Projectʹظ଴͢Δ͜ͱ

Slide 16

Slide 16 text

ΞδΣϯμ • ฐࣾͷBulkdataࣄ৘ • ߏஙͨ͠Embulkج൫ • ଍Γͳ͔͔ͬͨΒิͬͨػೳ̑ͭ • Embulk Projectʹظ଴͢Δ͜ͱ

Slide 17

Slide 17 text

ฐࣾͷBulkdataࣄ৘

Slide 18

Slide 18 text

HDFS -> Vertica

Slide 19

Slide 19 text

ฐࣾͷBulkdataࣄ৘ • ݎ࿚ͳϩάճऩج൫ • શͯͷαʔϏεͷϩά͸HDFSʹ֨ೲ͞ΕΔ • ϑΥʔϚοτ΋౷Ұ͞Ε͍ͯΔ • TSV + JSONͷࠞ߹ϑΥʔϚοτ

Slide 20

Slide 20 text

ฐࣾͷBulkdataࣄ৘ • ෼ੳͰ࢖༻͢ΔओཁετϨʔδ͸Vertica • HDFSͷϩάΛͦͷ··Verticaʹ֨ೲ͍ͨ͠ • ͨͩ͠streaming insert͸Ͱ͖ͳ͍… • Vertica͸ߴස౓ͳσʔλ౤ೖʹඇৗʹ ऑ͍

Slide 21

Slide 21 text

Vertica? •ྻࢦ޲ܕߴ଎ूܭσʔλϕʔε •঎༻ιϑτ΢ΣΞϥΠηϯε(1TB·Ͱ͸ແྉ) •ඇৗʹ๛෋ͳ෼ੳؔ਺ •twitter΍facebookͰ΋࢖ΘΕͯΔ •ฐࣾͷ෼ੳ؀ڥͷओ࣠

Slide 22

Slide 22 text

ߴ·Δχʔζ • GCS, S3, BigQueryͳͲcloud storageͷར༻֦େ • HDFS -> Vertica ͚ͩͰ͸ͳ͍Bulkloadχʔζ

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Embulk

Slide 25

Slide 25 text

͜Ε͕΄͔ͬͨ͠Μ΍ʂ

Slide 26

Slide 26 text

ͬͦ͘͞ಋೖ

Slide 27

Slide 27 text

ΞδΣϯμ • ฐࣾͷBulkdataࣄ৘ • ߏஙͨ͠Embulkج൫ • ଍Γͳ͔͔ͬͨΒิͬͨػೳ̑ͭ • Embulk Projectʹظ଴͢Δ͜ͱ

Slide 28

Slide 28 text

Wrapper ߏஙͨ͠Embulkج൫

Slide 29

Slide 29 text

σʔλ֨ೲ·ͰͷྲྀΕ

Slide 30

Slide 30 text

Wrapper 1. GoogleSpreadSheet͔Β εΩʔϚ৘ใͱBulkloadλΠϓΛநग़

Slide 31

Slide 31 text

Wrapper 2. Ϋϥελ৘ใ΍εέδϡʔϧ৘ใͱ ඥ෇͚ͯMySQLʹ֨ೲ

Slide 32

Slide 32 text

Wrapper 3. εέδϡʔϧʹैͬͯRedis΁ Enqueue

Slide 33

Slide 33 text

Wrapper 4. ඥͮ͘BulkloadλΠϓ΍ɺΫϥελ ৘ใ͔Βconfig.ymlΛࣗಈੜ੒

Slide 34

Slide 34 text

Wrapper 5. Embulk WrapperΛkick

Slide 35

Slide 35 text

Wrapper 6. ֨ೲઌͷσʔλΛ࡟আ

Slide 36

Slide 36 text

Wrapper 7. Embulk run

Slide 37

Slide 37 text

Wrapper 8. ॲཧ݁ՌΛMySQLʹอଘ

Slide 38

Slide 38 text

Wrapper 9. ݁Ռදࣔ

Slide 39

Slide 39 text

ৄࡉͳ࣮૷ʹؔͯ͠ ؾʹͳΔํ͸࠙਌ձͰʂ

Slide 40

Slide 40 text

ࠓ೔࿩͢෦෼

Slide 41

Slide 41 text

Wrapper ࠓ೔࿩͢෦෼

Slide 42

Slide 42 text

Wrapper ࠓ೔࿩͢෦෼ ͳΜͰspreadsheet࢖ͬͯΔͷʁ

Slide 43

Slide 43 text

Wrapper ࠓ೔࿩͢෦෼ Job Queue؅ཧͲ͏΍ͬͯΔͷʁ

Slide 44

Slide 44 text

Wrapper ࠓ೔࿩͢෦෼ ͳΜͰwrapper͔·ͯ͠Δͷʁ

Slide 45

Slide 45 text

Wrapper ࠓ೔࿩͢෦෼ embulkʹ໰୊͸ͳ͔ͬͨͷʁ

Slide 46

Slide 46 text

ͳͲʹ͍ͭͯ࿩͠·͢

Slide 47

Slide 47 text

ΞδΣϯμ • ฐࣾͷBulkdataࣄ৘ • ߏஙͨ͠Embulkج൫ • ଍Γͳ͔͔ͬͨΒิͬͨػೳ̑ͭ • Embulk Projectʹظ଴͢Δ͜ͱ

Slide 48

Slide 48 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 49

Slide 49 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 50

Slide 50 text

଍Γͳ͍͜ͱ̍

Slide 51

Slide 51 text

YAML؅ཧ

Slide 52

Slide 52 text

YAML؅ཧ • Embulkͷ࢓༷͚ͩͲɻɻɻ • 1ͭͷBulkloadʹରͯ͠1ͭͷYAMLϑΝΠϧ

Slide 53

Slide 53 text

ฐࣾͷࣄ৘ • εΩʔϚ৘ใ͸ΞφϦετ͕ఆٛ • εΩʔϚ৘ใҎ֎ͷઃఆ͸෼ੳج൫͕؅ཧ => ͻͱͭͷconfig.ymlੜ੒ʹෳ਺ਓ͕ؔΘΔ͜ͱʹ…

Slide 54

Slide 54 text

ฐࣾͷࣄ৘ • ର৅ͷBulkload͸1αʔϏε͋ͨΓ20~50 • ৗʹ20Ҏ্ͷαʔϏε͕ฒྻͰՔಇ͠ɺҠΓม ΘΓ΋ܹ͍͠ => େྔͷyaml͕ੜ੒͞Εͯ؅ཧ͕ͭΒ͍(઀ଓ৘ใͱ͔)

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

ج൫؅ཧ ΞφϦετ؅ཧ

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

൒෼͘Β͍ʹͳͬͨ

Slide 59

Slide 59 text

Α͘มΘΔ ΄ͱΜͲมΘΒͳ͍

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

݁ہΑ͘มΘΔͷ͸ εΩʔϚ৘ใ͘Β͍

Slide 62

Slide 62 text

ͭͬͨ͘ • Embulk Config Generator & UI • ઀ଓ৘ใΛDBͰҰݩ؅ཧ • BulkloadύλʔϯΛந৅Խͯ͠DBͰ؅ཧ • εΩʔϚ৘ใ͸spreadsheetͰ؅ཧ

Slide 63

Slide 63 text

ͭͬͨ͘ • config.yml͸࣮ߦ௚લʹੜ੒͞ΕΔͷͰ࢓༷ม ߋͳͲʹ΋ॊೈʹରԠͰ͖Δ • ࢒೦ͳ͕Βguess͸࢖͑ͳ͘ͳΓ·ͨ͠ɻ • ಛʹࠔͬͯͳ͍ • ϩάΛఆٛͨ͠ਓ͕εΩʔϚΛఆٛͯ͠Δ

Slide 64

Slide 64 text

Embulk Projectʹظ଴͢Δ͜ͱ • େن໛ར༻ʹͱͬͯͷyaml • fileͰશͯΛ؅ཧ͢Δͷ͸ͭΒ͍ • ಉ͡yaml಺Ͱ΋؅ཧऀ͕ҟͳΔ • ಉ͡yaml಺Ͱ΋มߋස౓͕ҟͳΔ

Slide 65

Slide 65 text

Embulk Projectʹظ଴͢Δ͜ͱ • templateػೳ͚ͩͰͳ͘ɺΑΓଟ࣍ݩʹ؅ཧͰ ͖Δ࢓૊Έ͕ඞཁͳͷͰ͸ʁ • ҰํͰɺ݁ہɺۀ຿ϑϩʔʹΑΔͷͰΈΜͳࣗ ࡞͢Δͷ͔΋͠Εͳ͍ͱ΋ࢥͬͯΔ

Slide 66

Slide 66 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 67

Slide 67 text

଍Γͳ͍͜ͱ2

Slide 68

Slide 68 text

୯ମॲཧ ϑϨʔϜϫʔΫ

Slide 69

Slide 69 text

Embulk͸Bulkload͢Δ͚ͩ • ઃఆ௨Γʹɺ͋Δσʔλιʔε͔Βɺ͋Δσʔ λιʔε΁σʔλΛϩʔυ͢Δ • Input / Outputͷঢ়ଶʹ͸ڵຯ͕ແ͍

Slide 70

Slide 70 text

͔ͩΒEmbulkͷ୲อ͢Δႈ౳ੑ ͸͜͏ͳΔ • 1ճ࣮ߦ͠Α͏ͱͨ͠Bulkloadʹର͢Δႈ౳ੑΛ ୲อ͠Α͏ͱ͢ΔʢPlugin࣍ୈʣ • ౰વ͚ͩͲɺ֨ೲઌͷσʔλͷ੔߹ੑΛอͭ΋ ͷͰ͸ͳ͍

Slide 71

Slide 71 text

ฐࣾͷࣄ৘ • HDFS͔ΒVertica΁ͷ֨ೲ͸ৗʹΧϥϜΛߜΔʢ͓ ۚΣ…ʣ • ෳࡶͳKPI͕ݟͨ͘ͳͬͨ࣌ʹΧϥϜ௥ՃΛߦ͏ • HDFS্ͷσʔλ͕ඞཁʹͳͬͨλΠϛϯάͰ σʔλͷ࠶ϩʔυ͕ൃੜ => ࣄલʹ֨ೲൣғͷσʔλ࡟আ͕ඞཁ

Slide 72

Slide 72 text

ฐࣾͷࣄ৘ • File֨ೲ͕׬͍ྃͯ͠Δ͔Ͳ͏͔Λ `_SUCCESS` ͱ͍͏ϑΝΠϧΛಉ֊૚ʹஔ͘͜ͱͰ؅ཧ • `_SUCCESS` ͕ͳ͚Ε͹ॲཧதͷσʔλͱ ೝࣝ͞ΕΔ => ࣄલʹϑΝΠϧͷଘࡏ֬ೝͱࣄޙʹϑΝΠϧͷ put͕ඞཁ

Slide 73

Slide 73 text

ͱ͍͏͔ • ձࣾʹΑͬͯϧʔϧ͸ҧ͑Ͳɺࣄલॲཧࣄޙॲ ཧ͸ઈର͍Δ͸ͣ

Slide 74

Slide 74 text

ͭͬͨ͘ • Embulk Wrapper • Embulkʹ଍Γͳ͍ࣄલɾࣄޙͷ୯ମॲཧΛ αϙʔτ͢ΔWrapper • YAMLͷઃఆϑΝΠϧʹج͍࣮ͮͯߦ͞ΕΔ • action × storage ͱ͍͏୯ҐͰpluginΛॻ͘

Slide 75

Slide 75 text

ͪΐͬͱ͚ͩ Embulk Wrapper঺հ આ໌༻ͷYaml͕ ؒԆͼ͢ΔͷͰ ΞϯΧʔ࢖͍·͢ɻ

Slide 76

Slide 76 text

actionͱ͍ actionͱ͍͏୯ҐͰ ॲཧΛఆٛ

Slide 77

Slide 77 text

actionΛarrayͰఆٛ͠ γʔέϯγϟϧʹ࣮ߦ

Slide 78

Slide 78 text

embulkͷ࣮ߦ

Slide 79

Slide 79 text

࡞ͬͨActions • vertica#delete • vertica#check_null • vertica#glance (νϥݟ) • s3#poll • s3#remove • ….

Slide 80

Slide 80 text

Embulk Projectʹظ଴͢Δ͜ͱ • Embulkʹ͸୯ମॲཧϑϨʔϜϫʔΫඞཁෆՄܽ • ґଘؔ܎ͷͳ͍୯ମॲཧΛߦ͏ϓϥάΠϯػߏ ͕Embulkʹ಺ଂ͞ΕΕ͹ɺEmbulkͷΈͰߦ͑Δ ͜ͱͷ෯͕͔ͳΓ޿͕ΔͷͰ͸ͳ͍ͩΖ͏͔ʁ

Slide 81

Slide 81 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 82

Slide 82 text

଍Γͳ͍͜ͱ3

Slide 83

Slide 83 text

δϣϒΩϡʔ

Slide 84

Slide 84 text

Embullk ͷ࣮ߦΛ Ͳ͏؅ཧ͠Α͏͔ • େن໛ར༻͢ΔͳΒδϣϒ؅ཧͷ࢓૊Έ͕ඞਢ • ౰વ͚ͩͲ Embulk ʹͦΜͳػೳ͸ͳ͍ • ࣗ෼Ͱ࡞Δ͔طଘͷผͷԿ͔Λ࢖͏͔͠ແ͍

Slide 85

Slide 85 text

݁ہɺͭͬͨ͘ • Redis/Sidekiq Λ࢖ͬͨfifoͳδϣϒΩϡʔ • εέδϡʔϧ࣮ߦ͸ɺεέδϡʔϥ͕࣌ؒʹͳΔ ͱ job Λ enqueue • Adhoc࣮ߦ͸ɺϢʔβʔ͕δϣϒ࣮ߦϘλϯΛԡ ͢ͱ job Λ enqueue => ͍͢͝ී௨ͷδϣϒΩϡʔ͆

Slide 86

Slide 86 text

Workerαʔόʔ(Embulk࣮ߦ؀ڥ) • 1αʔόʔ͋ͨΓɺEmbulk 1 process • Embulk͸CPUΛ࢖͑Δ΄Ͳߴ଎ʹͳΔͨΊ • 24 CPU, memory 60 GB αʔόʔ 6୆Ͱฒྻ࣮ߦ

Slide 87

Slide 87 text

ࠓͷͱ͜Ζࠔͬͯͳ͍ͷ͕ͩ • ഉଞॲཧ͕ͪΌΜͱͰ͖͍ͯͳ͍ • ༏ઌॲཧ͕ग़͖ͯͦ͏ • graceful restartͰ͖ͯͳ͍ • … => Ұॠߟ͑Δ͚ͩͰग़͖ͯͦ͏ͳ໰୊͕͍ͬͺ͍

Slide 88

Slide 88 text

͜Ε͸ྲྀੴʹEmbulkʹ͸ ظ଴͸ͯ͠ͳ͍ • ॻ͍͚ͨͲEmbulkʹ଍Γͳ͍͜ͱͱ͍͏ΑΓ Batch؀ڥʹ଍Γͳ͍͜ͱ͔ͬͯΜ͡Ͱͨ͠ • ͱ͸͍͑ɺEmbulkΛར༻͢Δ্Ͱඞཁͳ͜ͱͳ ͷͰϕετϓϥΫςΟε୳ͯ͠·͢

Slide 89

Slide 89 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 90

Slide 90 text

଍Γͳ͍͜ͱ4

Slide 91

Slide 91 text

ฒྻ਺੍ޚ

Slide 92

Slide 92 text

Embulkͷฒྻ਺੍ޚ • ݱঢ়LocalExecutor͸InputͷϑΝΠϧ਺Ͱશମͷ ฒྻ਺Λ੍ޚ͍ͯ͠Δ • FileInputPluginܧঝͰͳ͚Ε͹ฒྻ਺͸جຊతʹ 1ͭ

Slide 93

Slide 93 text

ฐࣾͷࣄ৘ • HDFSʹ֨ೲ͞Ε͍ͯΔϑΝΠϧ͸1ϑΝΠϧ͕ ڊେ • namenodeͷϝλ৘ใΛۃྗগͳ͘͢ΔͨΊ • ϑΝΠϧ͕෼͔Ε͍ͯͳ͍ͨΊɺϚϧνεϨου Ͱಈ͔ͣɺCPUΛશવ࢖͑ͳ͍

Slide 94

Slide 94 text

ͭͬͨ͘ • embulk-input-hdfs • ಉ͡ϑΝΠϧͷinput streamΛฒྻ਺෼ੜ੒ ͠ɺඞཁൣғͷΈread͢Δ࢓૊Έ • configʹॻ͔Εͨ਺ʹϑΝΠϧΛ෼ׂ • ೚ҙͷฒྻ਺Ͱembulk࣮ߦ͕Մೳʹʂ

Slide 95

Slide 95 text

͜ΕͰCPU ϑϧʹ࢖͑Δͧʂ

Slide 96

Slide 96 text

ͱࢥͬͨΒ໰୊͕ • OutputͰ͋ΔVertica • Session਺͕૿͑͗͢ΔͱVertica͕ෆ҆ఆʹ • ࠷େ20͘Β͍ʹߜͬͯ͘ΕͱVerticaνʔϜ ͔Βґཔ • InputͰฒྻ਺نఆ͞ΕΔͷʹͲ͏͢Ε͹͑͑Μ ΍Ͱ

Slide 97

Slide 97 text

ͭͬͨ͘ • embulk-output-vertica • ॲཧલʹσʔλ֨ೲ༻ͷthreadΛผ్੾Δ • ֤εϨου͸σʔλ֨ೲ༻ͷthreadʹpageΛ enqueue • σʔλ֨ೲ༻thread͸pageΛdequeue͠ίϐʔͯ͠ ͍͘ • ͜ΕͰ1ճͷίϐʔͰ1session͔͠࢖ΘͣʹࡁΉͧʂ

Slide 98

Slide 98 text

Embulk Projectʹظ଴͢Δ͜ͱ • ฒྻ਺੍ޚ͸΍ͬͺΓExecutorʹ΍ͬͯ΄͍͠ • input / filters / outputόϥόϥʹઃఆ͍ͨ͠ • https://github.com/embulk/embulk/issues/ 232 • ϑΝΠϧ෼ׂͷAPIͱ͔FileInputPluginͷAPIͱ͠ ͍͍ͯ͋ͬͯΜ͡Όͳ͍͔͠Β

Slide 99

Slide 99 text

଍Γͳ͍͜ͱϥΠϯφοϓ 1. YAML؅ཧ 2. ୯ମॲཧϑϨʔϜϫʔΫ 3. δϣϒΩϡʔ 4. ฒྻ਺੍ޚ 5. ଓ͖͔Β࣮ߦ

Slide 100

Slide 100 text

଍Γͳ͍͜ͱ5

Slide 101

Slide 101 text

ଓ͖͔Β࣮ߦ

Slide 102

Slide 102 text

Embulkͷػೳͱͯ͠ͷ ʮଓ͖͔Β࣮ߦʯ • Embulk࣮ߦ࣌ʹɺ࣍ʹEmbulkΛ࣮ߦ͢Δ࣌ʹ࢖ ༻͢ΔconfigϑΝΠϧΛੜ੒͓ͯ͘͠ • Input Plugin୯ମͷػೳͱͯ͠ఏڙ͞Ε͍ͯΔ embulk run /path/to/next-config.yml \ -o /path/to/next-config.yml

Slide 103

Slide 103 text

ฐࣾͷࣄ৘ • HDFS্ͷϑΝΠϧʹσʔλ͕௥ه͞Εͯߦ͘͜ ͱ͕ଟ͍ • ϑΝΠϧύε͸มΘΒͳ͍ɺͰ΋ɺඞཁͳσʔ λ͕૿͑ͯΔ • σʔλͷ಺༰͔ΒFilter͢Δ͔Ͳ͏͔൑அ͢ Δඞཁ͕͋Δ

Slide 104

Slide 104 text

΍Γ͔ͨͬͨ͜ͱ • Outputઌ(Vertica) ΁֨ೲ͞Ε͍ͯΔσʔλΛ֬ ೝ͠ɺInputσʔλΛFiltering͢Δ • SELECT max(id) FROM table; ͷ݁Ռ͔Β • embulk-filter-row ͷconditionੜ੒

Slide 105

Slide 105 text

ͭͬͯ͘ͳ͍ • ୯ମॲཧϑϨʔϜϫʔΫͱ͸ҧ͍ɺॲཧؒͷґ ଘؔ܎͕ଘࡏ͢ΔͨΊɺͬ͘͞ͱ͸Ͱ͖·ͤΜ Ͱͨ͠ٽ • Embulk WrapperͰର৅ൣғΛࣄલʹফͯ͠શͯ ϩʔυ͢Δͱ͍͏ྗٕͰରԠͨ͠

Slide 106

Slide 106 text

Embulk Projectʹظ଴͢Δ͜ͱ • ΄Μͱ͏ͷҙຯͰ੔߹ੑΛอͭʹ͸Outputଆͷ σʔλΛݟʹߦ͘ඞཁ͕͋Δ • ࣮ࡍʹॻ͜͏ͱ͢ΔͱEmbulkͷྖҬΛ௒͑Δ • ࡢ೔ฉ͍͚ͨͲʮdigdagʯͱ͍͏πʔϧ͕ग़ΔΒ ͍͠ΐ • https://github.com/treasure-data/digdag-docs

Slide 107

Slide 107 text

·ͱΊ

Slide 108

Slide 108 text

·ͱΊ • Embulkಋೖͯ͠ϝονϟḿͬͯΔ • ৽͍͠σʔλιʔε͕ग़͖ͯͯ΋ίετ௿͘ ಋೖͰ͖Δʂ

Slide 109

Slide 109 text

·ͱΊ • ͨͩɺ࣮ࡍproductionӡ༻͠Α͏ͱ͢Δͱࡉ͔͍ ͜ͱ͕ؾʹͳͬͯ͘Δ • YAMLͷ࿩ͱ͔ • δϣϒΩϡʔͷ࿩ͱ͔ • લॲཧɺޙॲཧͲ͏͢Δͷͱ͔

Slide 110

Slide 110 text

·ͱΊ • ࣮ࡍɺEmbulk͚ͩͰղܾ͢ΔΑ͏ͳ࿩Ͱ͸ͳ͍ • ΈΜͳͰ஌ݟͨΊͯϕετϓϥΫςΟε୳͍ͯ͠ ͖·͠ΐ͏ • AdventCalendarۭ͍ͯΔͷͰॻ͖·͠ΐ͏

Slide 111

Slide 111 text

͝ਗ਼ௌ͋Γ͕ͱ͏ ͍͟͝·ͨ͠ʂ