VertexAIで構築したMLOps基盤の取り組み

by CyberAgent

Slide 1

Slide 1 text

ݪ ࿨ر 2021೥౓ ৽ଔೖࣾ גࣜձࣾCAM ػցֶशΤϯδχΞ @harappa80 @harappa80

Slide 2

Slide 2 text

ݪ ࿨ر 2021೥౓ ৽ଔೖࣾ גࣜձࣾCAM ػցֶशΤϯδχΞ @harappa80 @harappa80

Slide 3

Slide 3 text

1. ࠓ೔࿩͢͜ͱ 2. CAMʹ͍ͭͯ 3. MLOpsͱ͸ʁͳͥඞཁʁ 4. CAMͰͷٕज़બఆ 5. VertexAIͷ֓ཁ 6. MLOpsج൫ͷߏ੒ 7. ӡ༻ͯ͠Έͯͷར఺ 8. ։ൃ/ӡ༻Ͱࠔ೉ͳ఺ 9. ج൫ͷվળ఺ 10. ·ͱΊ ໨࣍

Slide 4

Slide 4 text

1. ࠓ೔࿩͢͜ͱ ● MLOpsͱ͸ͳʹ͔ ● ͲͷΑ͏ͳߟ͑ͰMLOpsج൫Λߏங͔ͨ͠ ● VertexAIΛ׆༻ͯ͠ߴ଎Ͱج൫Λߏஙͨ͠ࣄྫ ● ૝ఆ͢Δࢹௌऀ • ͜Ε͔ΒMLOpsΛಋೖ͠Α͏ͱ͍ͯ͠Δํ • VertexAI΁ͱҠ؅Λݕ౼͍ͯ͠Δํ

Slide 5

Slide 5 text

2. CAMʹ͍ͭͯ ● αΠόʔΤʔδΣϯτάϧʔϓ࠷ݹࢀͷࢠձࣾ ● ΤϯλϝɺϥΠϑελΠϧɺϏδωεόϥΤςΟϝσΟΞΛओ࣠ ͱͨ͠ଟ਺ͷWebαʔϏεΛల։ ●ձࣾ঺հ

Slide 6

Slide 6 text

2. CAMʹ͍ͭͯ ● Fensi Platformͱ͍͏ࣾ಺ͷαʔϏε։ൃج൫Λ׆༻ ●αʔϏε։ൃʹඞཁͳڞ௨ػೳΛఏڙ   → αʔϏε։ൃΛߴ଎Խ ●౷Ұͨ͠ϩάϑΥʔϚοτͰσʔλΛऩू   → ϨϙʔςΟϯά΍ϨίϝϯυΛڞ௨ͷ࢓૊ΈͰಋೖ ●σʔλपΓͷಛ௃

Slide 7

Slide 7 text

3. MLOpsͱ͸ʁͳͥඞཁʁ

Slide 8

Slide 8 text

MLOpsͱ͸ ●MLOpsͱ͸DevOpsͱMachine Learningͷ଄ޠ ●ػցֶशγεςϜͷϓϩμΫτ΁ͷಋೖͱ  ͦͷ؅ཧΛԁ׈/ਝ଎Խ͢ΔͨΊͷٕज़ͱΞϓϩʔν

Slide 9

Slide 9 text

ͳͥMLOps͕ඞཁʁ

Slide 10

Slide 10 text

ػցֶशγεςϜΛܧଓతʹӡ༻͠ ϢʔβʔʹՁ஋Λಧ͚ΔͨΊ

Slide 11

Slide 11 text

ϩʔΧϧ։ൃ/ Notebook MLOpsج൫͕ͳ͍৔߹ • ൃੜ͠͏Δোน • ͲͷΑ͏ʹͯ͠ϓϩμΫτ΁ͱಋೖ͢Δʁ • ϓϩμΫτಋೖ·Ͱʹ͕͔͔࣌ؒΔ • ϩʔΧϧݕূ࣌ͱಉ౳ͷਫ਼౓Λຊ൪Ͱอূग़དྷͳ͍ • ೔ʑͷϢʔβʔΠϯλϥΫγϣϯΛܧଓతʹ࠶ֶशग़དྷͳ͍ • Ϟσϧͷਫ਼౓ྼԽʹؾ͕෇͔ͳ͍ DS MLϞσϧ

Slide 12

Slide 12 text

ػցֶशϞσϧΛ։ൃ͢Δ໨త ˚ ߴਫ਼౓ͳϞσϧΛ࡞ΔͨΊ ˕ αʔϏεʹಋೖ͠ɼϢʔβʔʹՁ஋Λܧଓతʹಧ͚ΔͨΊ →͜ΕΛαϙʔτ͢Δͷ͕MLOpsج൫ͷ໾ׂ

Slide 13

Slide 13 text

Ϗδωε্Ͱ͸ϓϩμΫτ΁ͷಋೖΛҙࣝͨ͠ ػցֶशϞσϧͷ։ൃ͕ॏཁ

Slide 14

Slide 14 text

● ػցֶशϞσϧΛֶश͢Δίʔυ͸ɼػցֶशγεςϜΛߏ੒͢Δཁૉͷ͏ͪͷҰ෦෼ ● ֶशҎ֎ʹ΋ଟ͘ͷॲཧ͕ඞཁͰ͋ΓɼෳࡶʹೖΓ૊ΜͰ͍Δ ●MLOpsج൫ͷ੹຿͸͜ΕΒΛ౷߹తʹ؅ཧ/ӡ༻Մೳͳ؀ڥΛఏڙ͢Δ͜ͱ ࢀরݩɿHidden Technical Debt in Machine Learning Systems MLOpsج൫ͷ੹຿

Slide 15

Slide 15 text

● ԼهͷΑ͏ͳίϯϙʔωϯτΛਖ਼͔֬ͭਖ਼͍͠ॱংͰ೔ʑܧଓతʹ࣮ߦ͢Δඞཁ͕͋Δ ● Ϟσϧͷਫ਼౓؂ࢹ΍ɼޙʹੜ੒෺Λ௥੻ग़དྷΔ࢓૊Έ΋ඞཁ  ࢀরݩɿTFX: A TensorFlow-Based Production-Scale Machine Learning Platform MLOpsج൫ͷ੹຿

Slide 16

Slide 16 text

MLOpsج൫͕͋Δ৔߹ • ࣮ݱग़དྷΔ͜ͱ • DS͕։ൃͨ͠ϞσϧΛਝ଎ʹαʔϏεͱ΁ಋೖɾվળ • ܧଓతʹ࠷৽ͷσʔλͰϞσϧΛ࠶ֶश • ϞσϧؒͷA/Bςετ • Ϟσϧͷਫ਼౓ྼԽ΍ಛ௃ྔͷҟৗΛݕग़ DS MLϞσϧ MLOpsج൫ • ࠶ֶश • A/B Test

Slide 17

Slide 17 text

MLOps͸҆ఆͯ͠ػցֶशϞσϧΛӡ༻͢Δ্Ͱͷඞਢٕज़

Slide 18

Slide 18 text

4.CAMͰͷٕज़બఆ

Slide 19

Slide 19 text

CAMͰͷٕज़બఆ ●MLOpsΛ࣮ݱ͢ΔͨΊͷٕज़ɾϑϨʔϜϫʔΫ • OSSɿ • MLFlowɿ༰қʹ࣮ݧ؅ཧՄೳ • AirflowɿػցֶशʹಛԽͨ͠΋ͷͰ͸ͳ͍ • KubeflowɿεέʔϦϯά͢ΔɻKubernetesͷ؀ڥ͕ඞཁ • Ϋϥ΢υɿ • VertexAI(Google Cloud) • Sagemaker(AWS)  → ݁࿦ͱͯ͠ɼCAMͰ͸VertexAIΛ࠾༻/ ӡ༻

Slide 20

Slide 20 text

CAMͰͷٕज़બఆ ●VertexAIΛ࠾༻ͨ͠ཧ༝ 1. ૉૣ͘ج൫Λ࡞੒ग़དྷΔ͜ͱ  →ϑϧϚωʔδυͳαʔϏε 2. ࿈ܞ͕༰қͳ͜ͱ  →σʔλج൫͕GCPʹ͋ΔͨΊಉ؀ڥ͕ཧ૝ 3. Pipeline͕OSSͷKubeFlow Pipelinesͱޓ׵  →ެ։͞ΕͨίϯϙʔωϯτΛ࠶ར༻Մೳ 4. ֤ίϯϙʔωϯτͷϚγϯεϖοΫΛॊೈʹࢦఆՄೳ  →ֶश΍લॲཧʹ͸ϋΠεϖοΫͳϚγϯ͕ඞཁ

Slide 21

Slide 21 text

5.VertexAIͷ֓ཁ

Slide 22

Slide 22 text

5. VertexAIͷ֓ཁ ࢀরݩ: https://cloud.google.com/blog/ja/topics/developers-practitioners/new-ml-learning-path-vertex-ai • 2021೥5݄ʹGCP্ͰϦϦʔε • CAMͰ͸൒೥ޙʹຊ൪Քಇ։࢝ • AI Platform, AutoML, MLܥͷAPIαʔϏεʢVisionAPI, Natural Language APIͳͲʣ͕౷߹͞ΕͨϓϥοτϑΥʔϜ

Slide 23

Slide 23 text

● ʮֶश༻σʔλͷੜ੒ʙਪ࿦݁Ռͷग़ྗʯ·ͰΛΧόʔ͢Δඞཁੑ͕͋Δ ●ɹ֤छλεΫΛΧόʔ͢ΔαʔϏε͕໢ཏతʹଘࡏ 5. VertexAIͷ֓ཁ ●VertexAIͱMLϫʔΫϑϩʔͷରԠ

Slide 24

Slide 24 text

6.MLOpsج൫ͷߏ੒

Slide 25

Slide 25 text

6.MLOpsج൫ͷߏ੒ ●ج൫ͷશମਤ

Slide 26

Slide 26 text

6.MLOpsج൫ͷߏ੒ ᶃύΠϓϥΠϯίϯϙʔωϯτͷ࡞੒ • ػցֶशʹඞཁͳλεΫ͝ͱʹίϯςφΛ࡞੒ɼGCR΁Push • CloudBuildͰCI/CD؀ڥΛ੔උˠ։ൃ/ຊ൪؀ڥͰಉ༷ͷॲཧ • ෳ਺ͷϞσϧؒͰڞ௨Խɾ࠶ར༻͢Δ͜ͱΛҙࣝͯ͠։ൃ • ެ։͞ΕͨίϯϙʔωϯτΛ࠶ར༻΋Մೳ

Slide 27

Slide 27 text

6.MLOpsج൫ͷߏ੒ ᶄύΠϓϥΠϯఆٛͷ࡞੒ • ࡞੒͞Εͨίϯϙʔωϯτͷ࣮ߦॱংͱύϥϝʔλΛఆٛ • CloudBuildͰίϯύΠϧͯ͠ɼPipeline࣮ߦ༻ͷjsonΛGCS΁֨ೲ͓ͯ͘͠

Slide 28

Slide 28 text

6.MLOpsج൫ͷߏ੒ ᶅֶश༻σʔληοτͷ࡞੒ • BigQueryʹ஝ੵ͞ΕͨϩάΛݩʹಛ௃ྔΛ೔ʑੜ੒ • ੜ੒͞Εͨಛ௃ྔ͸VertexAI Feature Storeʹ֨ೲ • BigQuery + Feature StoreΛࢀরֶͯ͠श༻σʔληοτΛ  ఆظతʹ࡞੒͢Δ • CloudComposerͰఆظ࣮ߦ

Slide 29

Slide 29 text

6.MLOpsج൫ͷߏ੒ ᶆύΠϓϥΠϯͷఆظ࣮ߦ • CloudComposerͰVertexAI PipelineΛεέδϡʔϧ࣮ߦ • GCS΁֨ೲࡁͷPipeline࣮ߦ༻ͷjsonΛݩʹ࣮ߦ • GCR΁PushࡁͷίϯςφΛॱ࣮࣍ߦ • ੜ੒෺ʢֶशσʔλ/ධՁࢦඪ/ϞσϧͳͲʣ͸Metadataʹ֨ೲ • ϞσϧσϓϩΠͷͨΊʹModel RegistryʹPush

Slide 30

Slide 30 text

6.MLOpsج൫ͷߏ੒ ᶇBatch/Online༧ଌͷ࣮ߦ • VertexAI PredictionͰ༧ଌΛ࣮ߦ • ϞσϧʹରԠͨ͠༧ଌ༻ίϯςφΛ༻ҙ͢Δඞཁ͕͋Δ • Batch༧ଌɿ • ೖྗɿGCSʹ֨ೲࡁͷJson • ग़ྗɿBigQuery·ͨ͸GCSʹग़ྗ • ෳ਺ϚγϯͰͷฒྻॲཧ΋Մೳ • Online༧ଌɿ • ༧ଌΛฦ͢ΤϯυϙΠϯτ͕࡞੒͞ΕΔ • ϚγϯεϖοΫΛॊೈʹࢦఆՄೳ

Slide 31

Slide 31 text

6.MLOpsج൫ͷߏ੒ ●ج൫ͷશମਤ

Slide 32

Slide 32 text

7. ӡ༻ͯ͠ͷར఺ • ֤ίϯϙʔωϯτͷ࣮ߦதͷΈΠϯελϯε্ཱ͕ͪ͛Δ • ίετΛ཈͑Δ͜ͱ͕Մೳ • ੜ੒෺ͷ؅ཧ͕༰қ • ίϯϙʔωϯτͷInput/Output͑͞ఆٛ͢Ε͹ɼ  ҙࣝ͢Δ͜ͱͳ͘GCS&VertexAI Metadataʹอଘ͞ΕΔ • ϚγϯεϖοΫΛॊೈʹઃఆग़དྷΔ

Slide 33

Slide 33 text

8. ։ൃ/ӡ༻Ͱࠔ೉ͳ఺ • ίϯϙʔωϯτͷ࡞੒ํ๏ • Container Base:   Model΍MetricsͷܕʹະରԠͰ͋Γɼ࣮ߦ࣌ϝλσʔλ ʢexecutor_inputʣΛࢀরͨ͠ϋοΫతͳ࣮૷͕ඞཁ • Python-Function Base:  શͯͷܕʹରԠͯ͠Δ͕ɼPipelineఆٛͷίʔυʹ  ίϯϙʔωϯτ಺ͷॲཧΛશͯهड़͢Δඞཁ͕͋Δ →ίʔυྔ͕গͳ͍؆қతͳPipelineͰ͋Ε͹͓͢͢Ί ●ίϯϙʔωϯτͷίϯςφԽ࣌ʹະରԠͷੜ੒෺ͷܕ͕͋Δ

Slide 34

Slide 34 text

9. ج൫ͷվળ఺ • ֤ίϯϙʔωϯτʹର͢Δςετͷಋೖ • Skew/Driftͷݕग़ػೳͷ௥Ճ • VertexAI Model MonitoringΛݕ౼த

Slide 35

Slide 35 text

10. ·ͱΊ • MLOps͸҆ఆͯ͠ػցֶशϞσϧΛӡ༻͢Δ্Ͱͷඞਢٕज़ • VertexAIΛ׆༻͢Δ͜ͱͰߴ଎ʹج൫ΛߏஙՄೳ • ೖࣾ1೥໨ͷࣗ෼Ͱ͑͞΋൒೥Ͱߏங/ຊ൪ಋೖग़དྷͨ ΑΓৄࡉͳ಺༰ɿ • Vertex AIΛ׆༻ͨ͠MLOpsͷ࣮ݱʲલฤʳ • Vertex AIΛ׆༻ͨ͠MLOpsͷ࣮ݱʲޙฤʳ