Tweet
Tweet

Slide 1

Slide 1 text

KinesisͱLambdaͰͭ͘Δ
 Serverlessͳϩάج൫ Torgayev Tamirlan (@prog893) CyberAgent, Inc.

Slide 2

Slide 2 text

• 2018೥ αΠόʔΤʔδΣϯτ৽ଔೖࣾ • Cloud Technologies Advisor = (SA+SRE+DevOps+Infra) / 4 • AWSϝΠϯͰ༷ʑͳαʔϏεΛαϙʔτ • AWA: DB෮چߴ଎Խ (12h→55m)ɺ
 ECS EC2 εέʔϧΠϯอޢ • Torte্ཱͪ͛ • AWAɺλοϓϧ஀ੜɺCROSS MEɺREQU • ܭ13αʔϏε୲౰ • ޷͖ͳAWSͷαʔϏε: ECS ࣗݾ঺հ Torgayev Tamirlan
 @prog893

Slide 3

Slide 3 text

ࠓ೔࿩͢͜ͱ • KinesisɺKinesis+Lambdaͷબఆཧ༝ • ͜ΕΒΛར༻͍ͯ͠Δ͍͔ͭ͘ͷฐࣾαʔϏεͷ
 ϩΪϯάΞʔΩςΫνϟΛ঺հ • ࠷ޙʹͪΐͬͱͨ͠αϓϥΠζ͋Γ

Slide 4

Slide 4 text

ࠓ೔࿩͞ͳ͍͜ͱ • fluentdͷࡉ͔͍࿩ • Kinesis/Lambda/S3/ES/… is Կ • ঺հ͢ΔαʔϏεͷόοΫΤϯυͷৄࡉ (ݴޠɺߏ੒ɺetc.) • Internal BI΍Contents moderationج൫ͷৄࡉ

Slide 5

Slide 5 text

Disclaimer • fluentdΛࣺͯΔ࿩Λ͠·͕͢ɺfluentd͕ݏ͍ͱ͍͏Θ͚Ͱ͸ͳ͍ • Ή͠Ζ޷͖ • fluentdͷݕূɺߏங͕ฐࣾͰͷ࠷ॳͷ࢓ࣄͰɺ
 ͦΕ͕ͳ͚Ε͹ਖ਼ࣜೖࣾͰ͖͍ͯͳ͍͔΋͠Εͳ͍ • fluentd΍flumeͷΑ͏ͳϛυϧ΢ΣΞ͕ద੾Ͱ͋Δࣄྫ΋ͨ͘͞Μ͋Δ • ֤αʔϏεͷཁ๬ۦಈͰಈ͘ϩʔϧͱͯ͠ಇ͍͍ͯΔͷͰѱ͔͠Βͣ

Slide 6

Slide 6 text

How we did it before ैདྷͷϩάసૹύλʔϯ΍՝୊

Slide 7

Slide 7 text

• ֤αʔό/ίϯςφͰfluentdΛDaemon/Sidecarͱͯ͠৐ͤͯϩάసૹ • ͦͷDaemon/sidecar͕࠷ऴతͳ֨ೲઌ (Ҏޙsink) ʹ௚઀֨ೲ EC2 ΞϓϦ + fluentd daemon S3 Bucket ϩάͷอଘ Internal BI fluentd endpoint Contents Moderation fluentd endpoint ैདྷͷϩάసૹ: ௚઀֨ೲύλʔϯ

Slide 8

Slide 8 text

ैདྷͷϩάసૹ: ू໿ύλʔϯ • ֤αʔό/ίϯςφͰfluentdΛDaemon/Sidecarͱͯ͠৐ͤͯϩάసૹ • fluentd Aggregator (Active/Active৑௕Խ) ͰϩάΛू໿͠ɺsinkʹసૹ ECS Service ΞϓϦ+fluentd
 ίϯςφ૬৐Γ EC2 fluentd aggregator S3 Bucket ϩάͷอଘ Internal BI Kinesis ingest Contents Moderation Kinesis ingest ΞϓϦέʔγϣϯ͕ࢮΜͰ΋ɺ
 ϩά͕ू໿αʔό·Ͱಧ͍͍ͯΕ͹0, TJOL͕ཷ·͍ͬͯͯ΋
 ू໿αʔό͸ͣͬͱ͍ΔͷͰ
 ϦτϥΠͰ͖Δ ☝

Slide 9

Slide 9 text

ैདྷͷ՝୊఺ • ϝϯςφϯείετ • εέʔϧΞ΢τɾεέʔϧΠϯͷखؒɺ֦ுίετ • ू໿αʔό૿ݮ΍sink௥ՃͰDaemon/SidecarͷίϯϑΟάमਖ਼ɺ
 σϓϩΠ͕ൃੜ • ܽଛϦεΫ • όοϑΝૹΓ͖ͬͯͳ͍αʔόͷεέʔϧΠϯͳͲʹΑΔαʔϏεΞ΢τ • ઃఆΛؒҧ͑ΔͱɺόοϑΝᷓΕͳͲͰܽଛɺࣄނൃੜ

Slide 10

Slide 10 text

Logging Challenges ϩάج൫ͷߏ੒ʹର͢Δཁ݅ͱ՝୊

Slide 11

Slide 11 text

Logging Challenges 1.৴པੑ 2.ϝϯςφϯείετ 3.εέʔϥϏϦςΟ 4.֦ுੑ 5.൚༻ੑ

Slide 12

Slide 12 text

Challenge: ৴པੑ • ϚονϯάαʔϏεͷΑ͏ͳϩάཁ͕݅ݫ͍͠αʔϏεͰ͸ɺ
 ΄ͱΜͲͷϩάʹର͢Δཁ͕݅׬શʹܽଛͳ͠ • Contents moderation༻ͷϩά΍ΞΫηεϩάΛؚΉ • ALBͷΞΫηεϩά͚ͩͰ͸μϝͩͬͨΓ • ܽଛͨ͠ࡍͷϦτϥΠػߏɺյΕͨϩά͕ྲྀΕͨͱ͖ͷٹग़ϑϩʔ͕ඞཁ • fluentdͷίϯϑΟάͰ΋Ͱ͖Δ͕ɺίϯϑΟά͕RubyͩΒ͚ʹͳΓɺ
 ϝϯςφϯείετ૿Ճ

Slide 13

Slide 13 text

Challenge: ϝϯςφϯείετ • Sidecar/Daemon fluentd (ͳͲͷϛυϧ΢ΣΞ) Λར༻͢Δͱɺ
 ͦΕΒͷ؅ཧɺϝϯςφϯε͕ඞཁ • ͞Βʹɺू໿ύλʔϯͰ͸ू໿αʔόͷ໘౗Λݟͳ͍ͱ͍͚ͳ͍ • ͕ͦ͜μϝʹͳΔͱશ෦μϝ • σΟεΫɺࢮ׆؂ࢹɺfluentdόοϑΝঢ়ଶɺηΩϡϦςΟύον΍
 OSߋ৽ͳͲ

Slide 14

Slide 14 text

Challenge: εέʔϥϏϦςΟ • ؆୯ʹεέʔϦϯάͰ͖ΔΑ͏ʹ͍ͨ͠ • ؆୯ = αʔόߏஙɺAnsibleྲྀ͠ΑΓ؆୯ • ϩάͷग़ྗ͕ஶ͘͠૿͑Δ͜ͱ͕૝ఆ͞Εͳ͍ͷͰɺ
 ΦʔτεέʔϦϯά͸ඞਢͰ͸ͳ͍ • σΟεΫ༰ྔ΍όοϑΝαΠζΑΓεϧʔϓοτϕʔεͰ
 ύϑΥʔϚϯεΛࢦఆ͍ͨ͠ • ͪ͜Βͷํ͕DevOps΍SRE͡Όͳ͍։ൃऀʹͱͬͯҰ൪Θ͔Γ΍͍͢

Slide 15

Slide 15 text

Challenge: ֦ுੑ • ֨ೲઌ͕૿͑Ε͹ɺ҆қʹߦ͑ΔΑ͏ʹ͍ͨ͠ • ྫ: KibanaΛ࢖ͬͨϩάՄࢹԽΛಋೖ͍ͨ͠ • ҆қ: • ΞϓϦέʔγϣϯͷσϓϩΠͳ͠ • ΞʔΩςΫνϟมߋͳ͠

Slide 16

Slide 16 text

Challenge: ൚༻ੑ • ͲͷΑ͏ͳαʔϏε΍ཁ݅ʹ΋ରԠͰ͖Δ • ࠶ݱੑ͕͋ΓɺಛఆͷαʔϏεɾཁ݅ʹґଘ͠ͳ͍ • χʔζʹ߹Θͤͯඍௐ੔Ͱ͖Δ • ࠶ར༻ & ࣗ༝ʹ૊Έ߹Θ͕ͤՄೳ

Slide 17

Slide 17 text

After *a lot* of 
 sleepless nights. . . ਺͖͑Εͳ͍຾Ε͵໷Λܦͯʜ

Slide 18

Slide 18 text

Serverless Logging with
 Kinesis and Lambda ,JOFTJTͱ-BNCEBΛ׆༻ͨ͠ϩάج൫ͷߏ੒

Slide 19

Slide 19 text

ͳͥ Kinesis • Buffer΍ʮ༰ྔʯͱ͍͏֓೦͕ͳ͍ • อ࣋ظؒɺεϧʔϓοτ͚ͩؾʹ͢Ε͹ྑ͍ • ௚ײతͳύϑΥʔϚϯεઃఆ • 1γϟʔυ͋ͨΓͷεϧʔϓοτ͕ܾ·͍ͬͯͯɺ
 εϧʔϓοτΛ૿΍͚ͨ͠Ε͹γϟʔυΛ૿΍͚ͩ͢ • reshard͕ࣗಈͰߦΘΕΔ

Slide 20

Slide 20 text

ͳͥ Kinesis + Lambda • Kinesis + Lambda Event Source MappingͰ͸ɺϦτϥΠ͕ࣗಈ • Lambda͕ࣦഊ͢Ε͹ɺಉ͡σʔλͰ࠶࣮ߦ • ͦͷσʔλ͕ࣦޮ OR Lambdaͷ࣮ߦ͕੒ޭͰऴྃ͢Δ·ͰϦτϥΠ
 (ͲͪΒ͔ૣ͍ํ) • Kinesis্ͷσʔλΛͲ͜·Ͱॲཧ͔ͨ͠ͷϙδγϣϯ؅ཧ΋͞ΕΔ • ࣗલͰ༻ҙ͢Δ΋ͷ͸σʔλͷॲཧ΍సૹ༻ͷ࢓૊Έ͚ͩ! ָνϯ!

Slide 21

Slide 21 text

ߏ੒ v1 • Kinesis Streams -> Lambda -> ֤छ sink (S3/ES/Internal BI etc.) • Kinesis΁ͷ֨ೲ͸αʔϏεͷཁ݅ʗঢ়ଶʹΑΔ • ૬৐Γfluentd + aws-fluent-plugin-kinesis • Kinesis APIͰΞϓϦέʔγϣϯͷίʔυ͔Β௚઀֨ೲ

Slide 22

Slide 22 text

ࣄྫ1: λοϓϧ஀ੜ ECS Service ΞϓϦ+fluentd
 ίϯςφ૬৐Γ EC2 fluentd aggregator KDS to Elasticsearch S3 Bucket ϩάͷอଘ Lambda lambda-kinesis-to-es Elasticsearch Service ϩάௐࠪ with Kibana fluentd + aws-fluent-plugin-kinesis KDS to BI Internal BI Kinesis ingest KDS to Contents Moderation Contents Moderation Kinesis ingest fluentd + aws-fluent-plugin-kinesis fluentd + aws-fluent-plugin-kinesis

Slide 23

Slide 23 text

ࣄྫ2: CROSSME EC2 ΞϓϦ KDS to BI Kinesis SDKసૹ Internal BI Kinesis ingest KDS to Contents Moderation Contents Moderation Kinesis ingest KDS to Elasticsearch Lambda lambda-kinesis-to-es Elasticsearch Service ϩάௐࠪ with Kibana Kinesis SDKసૹ Kinesis SDKసૹ Lambda lambda-kinesis-to-es S3 Bucket ϩάͷอଘ

Slide 24

Slide 24 text

ࣄྫ3: REQU S3 Bucket ϩάͷอଘ KDS to Elasticsearch Lambda lambda-kinesis-to-es Elasticsearch Service ϩάௐࠪ with Kibana KDS to Contents Moderation Lambda lambda-kinesis-to-s3 ECS Service ΞϓϦ Kinesis SDKసૹ Kinesis SDKస ૹ Contents Moderation Kinesis ingest

Slide 25

Slide 25 text

Logging Challenges ౴͑߹Θͤ 1. ৴པੑ: KinesisͰͷू໿ɺKinesis+LambdaͷϦτϥΠػߏ 2. ϝϯςφϯείετ: ServerlessɺFull-Managed 3. εέʔϥϏϦςΟ: Kinesisγϟʔυ௥ՃɺࣗಈϦγϟʔυ 4. ֦ுੑ: LambdaϞδϡʔϧΛ޷͖ʹ૊Έ߹ΘͤͰ͖Δ 5. ൚༻ੑ: ͲͷαʔϏεͷཁ๬ʹ΋ରԠͰ͖ΔLambdaϞδϡʔϧͷ։ൃ

Slide 26

Slide 26 text

Feedback • ʮू໿༻fluentd͕ࣺͯΒΕͯؾ͍͍࣋ͪɺ࢒ͬͨ΋ͷ΋ࣺ͍ͯͨʯ • ʮfluent-plugin-kinesisʹཔΓͨ͘ͳ͍ʯ • ʮfluent-plugin-kinesisͷόοϑΝઃఆͳͲconfͷ؅ཧ͕ΊΜͲ͍ʯ • ʮKinesis APIΛୟ͘ύλʔϯͰɺfluentd૬౰ͷ࣮૷Λͨ͘͠ͳ͍ʯ

Slide 27

Slide 27 text

Serverless Logging with
 Kinesis and Lambda ͞ΒʹਐԽͨ͠,JOFTJTͱ-BNCEBΛ׆༻ͨ͠ϩάج൫ͷߏ੒ *Better

Slide 28

Slide 28 text

Serverlessͳϩάج൫ͷߏ੒ • ׬શʹfluentdͳͲͷϛυϧ΢ΣΞ͞Α͏ͳΒύλʔϯ • ECS -> stdout -> CWL -> CWL Subscription Filters -> 
 Kinesis “router” -> Lambda forwarder -> Kinesis sinks/S3/ES • Internal BIπʔϧ΍Contents moderationج൫͸ɺͦΕͧΕͷ։ൃνʔϜͱ
 ࿈ܞ͠ɺ֤AWSΞΧ΢ϯτͷKinesisετϦʔϜ͔ΒಡΈࠐΉΑ͏ʹมߋ • EC2΋CloudWatch Agentར༻͢Ε͹࢖͑Δ

Slide 29

Slide 29 text

ࣄྫ4: ։ൃத৽نαʔϏεx2 S3 Bucket ϩάͷอଘ Lambda lambda-kinesis-to-es Elasticsearch Service ϩάௐࠪ༻ES+Kibana Lambda lambda-kinesis-to-s3 KDS router Internal BI Kinesis ingest CloudWatch Logs Lambda lambda-kinesis-forward KDS to BI KDS to Contents Moderation Contents Moderation Kinesis ingest ※ lambda-kinesis-forward͕ΫϦςΟΧϧύεʹͳΓɺ
 શͯͷΠϕϯτΛऔಘ͢Δඞཁ͕͋ΔͷͰEFOར༻͕Φεεϝ ECS Service ΞϓϦ CloudWatch Logs Subscription Filters

Slide 30

Slide 30 text

஫ҙ: 
 CloudWatch LogsͷPutLogEvents͸ߴ͍ 
 (Kinesis΋...҆͘͸ͳ͍) ࡳଋ Ͱ৴པੑΛങ͍ͬͯΔΑ͏ͳ΋ͷ ͨͩɺྫ͑͹ECSͷϩΪϯάυϥΠόͰKinesis΁ͷ֨ೲ͕Ͱ͖ΔΑ͏ʹ ͳͬͨͱͯ͠΋ɺKinesisͷΩϟύ΍ϩάपΓLambdaͷෆ۩߹ͳͲ͔Β ϩάΛकΔͨΊͷόοΫΞοϓ͕ཉ͍͠

Slide 31

Slide 31 text

About Baikonur OSS #BJLPOVS044ϓϩδΣΫτʹ͍ͭͯ

Slide 32

Slide 32 text

Baikonur OSS Project • Terraform Module΍֤छπʔϧͷڞ௨ԽϓϩδΣΫτ • GitHub.comɺTerraform Module Registry • ฐࣾͰ։ൃɺར༻͍ͯ͠ΔϞδϡʔϧΛॱ࣍OSSԽ • ฐࣾ಺: AWSϞδϡʔϧ23छྨ • ໊લͷ༝དྷɿόΠίψʔϧӉ஦ج஍ • ͦͷ··࢖͑Δɺ͙͢ల։Ͱ͖ΔϕετϓϥΫςΟε

Slide 33

Slide 33 text

Baikonur OSS modules • ຊ೔঺հͨ͠શͯͷLambda͸BaikonurͰ঺հ͞Ε͍ͯ·͢ • ָʹߏங͕Ͱ͖ΔTerraform Module΋ఏڙ • ฐࣾ಺Ͱ࢖͍ͬͯΔ΋ͷΛͦͷ··ެ։

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

൪֎: eden • ECS Dynamic Environment Manager = eden • ECSͷ։ൃ؀ڥΛಈతʹ࡞੒͢Δπʔϧ΋Baikonur OSSͰެ։ • ৄࡉ: http://bit.ly/awseden

Slide 37

Slide 37 text

Kinesis Caveats ,JOFTJTΛར༻͢Δʹ͋ͨͬͯ஫ҙ͢΂͖ͱ͜Ζ

Slide 38

Slide 38 text

Scaling • γϟʔυ਺มߋͰͰ͖ͳ͍͜ͱ (υΩϡϝϯςʔγϣϯ) : • 24࣌ؒͰ3ճҎ্ͷγϟʔυ਺มߋ • ݱࡏͷγϟʔυ਺ΑΓ2ഒΑΓଟ͍ɺ൒෼ΑΓগͳ͍γϟʔυ਺΁ͷมߋ • 500γϟʔυΑΓଟ͍γϟʔυ਺΁ͷมߋ • 500γϟʔυΑΓଟ͍γϟʔυΛ࣋ͭετϦʔϜͷεέʔϧμ΢ϯෆՄ • ΞΧ΢ϯτ୯Ґͷγϟʔυ਺ͷ্ݶͷಥഁ • શ෦্ݶ؇࿨Մೳ

Slide 39

Slide 39 text

Autoscaling • ΦʔτεέʔϦϯά͸ఏڙ͞Ε͍ͯͳ͍ • awslabs/amazon-kinesis-scaling-utils Λ࢖͑͹ΦʔτεέʔϦϯάͬΆ͍
 ͜ͱ͕Ͱ͖Δ • ࢲͷܦݧͰ͸ΦʔτεέʔϦϯά͕ඞཁʹͳͬͨ͜ͱ͸ͳ͍ • ॻ͖ࠐΈɺಡΈࠐΈࣦഊ਺ͷ؂ࢹΛ௥Ճ͠ɺ
 ҟৗݕ஌ΛτϦΨʔʹखಈͰγϟʔυ૿ՃͰ͖Ε͹े෼ • Capacity ExceededͰϦτϥΠ͢ΔϩδοΫ֤ॴͰ࣮૷ࡁΈ • ্هscaling-utilsΛPythonͰॻ͖௚ͯ͠কདྷతʹΦʔτεέʔϧ͢Δ͔΋

Slide 40

Slide 40 text

Read Capacity: transactions໰୊ • Kinesis+LambdaͰ͸ɺ1γϟʔυ͋ͨΓಉ͡Lambda͕
 ಉ࣌ʹ1ճͷΈ࣮ߦ͞ΕΔ • ಡΈࠐΈεϧʔϓοτʹ͸ɺ5 transactions/secͱ͍͏੍ݶ͕͋Δ • 1 Lambda࿈ܞ1γϟʔυ͋ͨΓ~1 transaction/sec͕ফඅ͞ΕΔ • ॲཧͰ͖Δ৽͍͠σʔλ͕ͳ͍͔ͷ֬ೝΛຖඵ࣮ߦ • 3 LambdaҎ্ར༻͍ͨ͠৔߹ɺ3ͭ໨Ҏ߱͸ EFO Pipes Ͱઐ༻εϧʔϓοτ֬อ • Enhanced Fan-out

Slide 41

Slide 41 text

Conclusion ૯ׅ

Slide 42

Slide 42 text

૯ׅ • ؾܰʹ֦ுՄೳͳϩάपΓΞʔΩςΫνϟΛ࣮ݱ • KinesisͱLambdaͷ׆༻ͰServerlessߏ੒ • ϝϯςφϯείετݮ • Kinesis + Lambda࿈ܞͰࣗಈϦτϥΠ • ࣮૷Ͱ͸ϦτϥΠػߏΛ࣮૷͢Δඞཁ͕ͳ͘ɺΤϥʔॲཧͷΈͰे෼ • ڞ௨ԽɺOSSԽʹΑͬͯಋೖίετݮ

Slide 43

Slide 43 text

Thank you for listening! ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠