AWS GlueͰ ϦϓϨʔεͯ͠Έͨ גࣜձࣾGunosy ։ൃຊ෦ ඿஍྄ี

͓·͑ͩΕ • @aibou • SREνʔϜͰܯ࡯ۀ΍ͬͯ·͢ • ϏοάσʔλະܦݧͰ͢ • Ξϝϑτ؍ઓ͕޷͖Ͱ͢

ۀ຿಺༰ • άϊγʔɾ޿ࠂαʔόͷӡ༻ • ࣗಈԽɾলྗԽΛ໨తʹ೔ʑ׆ಈ • ίʔυԽ(codenize.tools, terraform) • ʮ͜ΜʹͪΘʔ ܯ࡯ͷ΋ͷͰ͕͢ʔʯ • OpsWorks, Kinesis3ܑఋ • X-Ray࢖͍͍ͨͳ͊ɾɾɾ

໨࣍ • GunosyͰGlueΛಋೖͨ͠࿩ʢ͔͠΋ยखؒͰʣ • AthenaύʔςΟγϣϯ࡞੒όονΛGlueʹஔ׵ • ࣗલETLόονΛGlueʹஔ׵ • DynamoDBͷϑΥʔϚοτม׵ΛGlueͰ࣮ݱ

ᶃAthenaύʔςΟγϣϯ࡞੒ όονΛGlueʹஔ׵

άϊγʔͷ޿ࠂϩάͷྲྀΕ ʢͬ͘͟Γʣ CBUDITFSWFS &.3 4QBSL "1*TFSWFST PS "UIFOB ഑৴ީิ 'JSFIPTF

AthenaͱRedshiftͷ࢖͍෼͚ • Athena • ՄࢹԽ༻ • ԕ͍աڈ΍௕ظؒͷूܭͰར༻ • Redshift • ϩδοΫ༻ • ௚ۙͷσʔλ͔Β഑৴ީิͷੜ੒౳ • ετϨʔδ࢖༻཰ϑϧʹͳΔͱRedshiftೖΕସ͑ΔରԠ • ʹݹ͍σʔλΛࣺͯΔ(S3ʹόοΫΞοϓ͋Γ)

Athena • fluent-plugin-s3ͰϩάΛS3ʹΞοϓϩʔυ • tag͝ͱʹ෼཭(imp, click౳) • hive friendly ( /click/year=2017/... ) ͳkeyͰ • ఆ࣌όονͰADD PARTITION͢Δ • MSCK REPAIRͩͱ࣮͕͕͔͔֬ͩ࣌ؒΔ • MSCK REPAIR ≒ 2h, ADD PARTITION < 1s

Athena ύʔςΟγϣϯ࡞੒όον • ࣾ಺Ͱ͸Կނ͔ʮશࣗಈ͍͋΅͏ʯͱݺ͹Ε͍ͯΔ • CloudWatch Events + Lambda • Lambda͸Java੡ • codenize-tools/monosasi ͰDSLԽ • s3://bucket/path/to/year=2017/month=11/day=24/hour=18/ • ཁ๬͕͋Ε͹ίʔυެ։͠·͢ʢ͕ɺGlueͷํ͕ศརΑͶɾɾɾ • FirehoseϑΥʔϚοτʹ΋ରԠՄೳ

Glue DataCatalog΁ • Athena΍LambdaͰRate limit • ҙࣝͯͣ͠Β͞ͳ͍ͱ͕࣌ؒूத͢Δ • εΩʔϚ؅ཧͱCloudWatch Eventsͷ؅ཧਏ͍ • ݱঢ়45ςʔϒϧ͙Β͍ɻطʹ͠ΜͲ͍

to Glue Crawler & DataCatalog "1*TFSWFST "-5&35"#-& "%%1"35*5*0/

ॴײ • Lambda͕ෆཁʹͳͬͨʢ΍ͬͨͶʣ • 1ΫϩʔϥͰෳ਺ͷSource DataStoreʹରԠ • Ϋϩʔϥ͕ཚཱ͠ͳͯ͘ॿ͔Δ • ϩάྔ͕ଟ͍ͱΫϩʔϦϯάʹ͕͔͔࣌ؒΔ • ͍·ͷͱ͜Ζ౎౓6͔͔࣌ؒͬͯΔ

GlueͰͰ͖ͳ͔ͬͨ͜ͱ • ҟϦʔδϣϯͷAthenaʹAdd table • ઌड़ͷύʔςΟγϣϯ௥ՃόονͰରԠ • طଘͷAthena Tableʹରͯ͠DataCatalogͷద༻ • database/tableͷ࡞Γ௚͠ʢӨڹ͋ΔͷͰ·ͩʣ • Glueͷςʔϒϧ໊ͷࢦఆɾมߋ • ಛҟͳϑΥʔϚοτʹରԠෆՄ ཌ೔

ᶄࣗલETLόονΛGlueʹஔ׵

ϩάͱϚελʔͷJOIN • ϩάσʔλɿS3 ɹɹ Redshift • imp, click etc • ϚελʔσʔλɿRDS • campaign, creative etc • ʮAthenaɾRedshiftͰJOIN͍ͨ͠ʯ • => Digdag + EmbulkͰରԠ͍ͯͨ͠

Embulk + digdag (+ docker) "-5&35"#-& "%%1"35*5*0/

to Glue Crawler & Glue ETL PSFHPOSFHJPO UPLZPSFHJPO SFQMJDB ᶃ ᶄ ᶄͷNFUBEBUBͰ BEEUBCMF ᶃͷNFUBEBUBͰ &5-

Additional • statsςʔϒϧɿ೔ຖͷूܭࡁΈσʔλ • ςʔϒϧશߦͰ͸ͳ͘೔ຖͷσʔλ͚ͩUpload • ETLͷJobͰFilter transform classΛ࢖༻ TUBUTUBCMF TQBUIUPZFBSNPOUIEBZ

Filter transformer class def filter_function(dynamic_record):
 if dynamic_record["date"].strftime("%Y-%m-%d") == yesterday.strftime("%Y-%m-%d"):
 return True
 else:
 return False
 
 filtered0 = Filter.apply(frame=datasource0, f=filter_function, transformation_ctx="filtered0")

GlueͰΑ͔ͬͨͱ͜Ζ • ETL͕Glue͚ͩͰ׬݁͢Δ • Lambdaෆཁ • Digdag + Embulkෆཁ • αʔό(ECS)ෆཁ • ਓ͕ؒςʔϒϧΛྻڍ͠ͳ͍͍ͯ͘ • ࠓ·ͰLambdaͱ͔EmbulkͷઃఆϑΝΠϧͰશ෦ࢦఆͯͨ͠

GlueͰΑ͘ͳ͔ͬͨͱ͜Ζ • CrawlerͰ͸ͳ͘ETLଆͰfilter͍ͯ͠Δ • ൃߦ͞ΕΔSQL͸ SELECT * FROM hoge_stats; • CrawlerଆͰWHEREઃఆ͍ͨ͠ • 18࣌ؒͰऴΘΒͳ͍Job (3ԯϨίʔυ) • JobͷΫϩʔϯ͕Ͱ͖ͳ͍ʢΘΓͱٸ͗Ͱ΄͍͠ʣ • ͍ͭ͘΋ࣅͨΑ͏ͳ΋ͷΛ࡞Δͷʹख͕ؒ

ᶅDynamoDBͷϑΥʔϚοτม׵Λ GlueͰ࣮ݱ (ϦϓϨʔεͰ͸ͳ͍͚Ͳ)

DynamoDBͷσʔλΛ෼ੳ͍ͨ͠ • άϊγʔͷλϒฒͼ৘ใ • Ͳ͏͍ͬͨϢʔβ͕Ͳ͏͍͏λϒΛϑΥϩʔͯ͠Δ͔ • ಛఆ৚݅Ͱࣗಈ௥Ճ͞ΕΔλϒ • ਖ਼͘͠ػೳ͍ͯ͠Δ͔ • ͲΕ͙Β͍͍Δͷ͔ • ೔͝ͱʹूܭ͍ͨ͠

DynamoDBͷFull dumpͱFormat • DataPipelineͰFull dump • DynamoDB Stream͸
 ࠓޙ΍Δʢئ๬ʣ • σʔλهड़ࢠ -> JSON • ͦͷ··͸ਏ͍ • (ͳΜͱ͔ͯ͠) convert͢Δ { "Item": { "Age": {"N": "8"}, "Colors": { "L": [ {"S": "White"}, {"S": "Brown"}, {"S": "Black"} ] }, "Name": {"S": "Fido"}, "Vaccinations": { "M": { "Rabies": { "L": [ {"S": "2009-03-17"}, {"S": "2011-09-21"}, {"S": "2014-07-08"} ] }, "Distemper": {"S": "2015-10-13"} } }, "Breed": {"S": "Beagle"}, "AnimalType": {"S": "Dog"} }

Data Pipeline + Glue ETL Amazon DynamoDB %BUB1JQFMJOF w .BQBQQMZ w UP%'GSPN%'

ETLͷ࣮ݱʹ͋ͨͬͯ • Built-in Transformer class͚ͩͰͳΜͱ΋ͳΒͳ͍͕࣌͋Δ • ΑΓෳࡶͳFilterɾMap࣌ • toDF & fromDFΛ࢖͏ʢී௨ͷpysparkʣ • job bookmarkΛ࢖͏࣌͸ try & except ඞਢ • toDFͰίέΔʢύʔςΟγϣϯʹ·ͭΘΔ໰୊ͬΆ͍ • job bookmarkͷϢʔεέʔε͕·ͩΘ͔ͬͯͳ͍

ϋϚΓϙΠϯτ • Built-in TransformerΫϥεͷίʔϧόοΫؔ਺಺Ͱൃੜͨ͠Exception͸ѲΓͭͿ͞ΕΔ • Exception͕ൃੜͨ͠dynamic record͸ഁغ͞Εɺ࣍ͷϨίʔυ΁ॲཧ͕ҠΔ • σόοά͢ΔͳΒίʔϧόοΫؔ਺શମΛtry-except • RDSΛSource DataStoreʹ͢Δ৔߹͸Subnetʹ஫ҙ • ͍ΘΏΔʮLambda in VPC໰୊ʯ • ඞͣNAT Gateway͔VPC EndpointͰS3ʹΞΫηεͰ͖ΔΑ͏ʹ͓ͯ͘͠(Scriptͷ DL) • NWઃఆ൓өʹԿނ͔30෼͙Β͍͕͔͔࣌ؒΔ

Glueͷॴײ • ࣗલETLόονશ෦ΛGlueʹஔ͖׵͍͑ͨ • DataCatalogͱ͔࠷ߴ • ͕ɺݱঢ়ͩͱ৭ʑਏ͍ • ࣮ߦ࣌ؒɾಥવࢮɾ։ൃࠔ೉

Glueʹର͢Δཁ๬ • ԿΑΓ΋ߴ଎Խ • DynamoDBΛSource DataStoreͷର৅ʹ • JobεΫϦϓτͷόʔδϣϯ؅ཧ • JobͷσόοάΛ༰қʹ(Τϥʔ͕ѲΓͭͿ͞ΕΔ) • Built-in Transform Classʹ͍ͭͯυΩϡϝϯτͷॆ࣮ • Crawler & ETL jobͷ؂ࢹ΄͍͠ • ScalaͰॻ͔ͤͯ͘Ε • EMR(Spark)ͷࢿ࢈Λྲྀ༻͍ͨ͠