Argoによる機械学習実行基盤の構築・運用からみえてきたこと

2019/7/22 Cloud Native Days Tokyo 2019での河野・稲村の講演資料になります

Transcript

1. G A A Q Q R D Google Kubernetes Engine MR N S C T O D      
    2 # 9 @2 0 A 0 / # :. 01 0 / # / # . / # 19 @2:0 A 20 7 1:2 #
2. )0 • A 0 J i s o d pl8D CC A C J ot l J E ) • J C l s ber4 J E J B A I • R R M- c u l s dr s T n kh S R l/ 1 / IJ D CA ACAJ 1E AE AE R i k S 2 22 0 ) ( 0 ? @9 : u u R u R R u R R ) ( & R P R R P s g a dP m l sch P y y P
3. ( ) 3 1 ? @ 0 00 0 : 7 / > > - 2 5 3 015 38
4. • ? ? • ? C ? CT • U A ? A / ? C UWL • ? / ? C p • ? / ? C • D • U D 4 • k lGcWtrw Ff e ibl • G W • G T A ? A / ? C • P ys W • ? xv I • G W ys ( E I • - T • m jGdn • / F gka c o F m • I A 23 . • ? / ? C - • ? D ) • ? A - • ? / ? C - • ? - 4 3 5 1
5. • • • • ?A • E • - • -
6. 
   Argoproj is a collection of tools for getting work done with Kubernetes. • Argo Workflows - Container-native Workflow Engine • Argo CD - Declarative GitOps Continuous Delivery • Argo Events - Event-based Dependency Manager • Argo Rollouts(NEW!) - additional deployment strategies https://argoproj.github.io/argo/
7. O D CR W G 7 K D 7 A
8. L + - - - • arfi - - - - 8- + -, +? • arfi 8 gc or
   Wb n c M ehW • kp cfel arfi E cfel cfels w V kp tEd ca h v
10. )G E EGA ELH - 8 G H • ) EG FH 8H 8G8 ED E LEGA ELH • )G 8 H FFEG • )G 8 EG / G8L • F K DF E F H • 8G 8 H F8G8C GH • 1EEFH • 8G8C G 8 ED • ED ED8 H • C E H H F LEGA EL K • 3 G H F LEGA EL K • 3 H C C CE • HF D 3 H C & • 8D 8 ED • 0(H G HE G EG H G8 ED • , /EEAH DE 8 EDH 8D F • 8G 8 E ED E ECF LEGA EL • D • 8 D E G8 EDH DE H EGH • E C H F C G8 H D • 8G8 HC C H • 8 CED H FH • D E A G D E A G • G F H FH
      
11. 1 - - 1 H E
12. - 12 E
13. 1 3 
    
14. • A F E • c d • c - - - - - • We a • • c We a
15. 5 / 1 /
16. 2 6 C B C 2 C 2 11 Co B
17. 7 1 1 • c P • S a E • • A I • c •
18. 8 - - -1 - 1- -1 • e U D • ca k nt D • g c g c L • a EU D T • D C g g cP • p i r i L S I I • a EU • g T IP a - 1- -1 • DI g c • g - -- - L E
19. :9- 8. - 2 / 02 10 • 9 9 1
20. 0 :9- 8. - 2 / 02 10 • 0 2 •  
      8 L l I m k 8 S m g l j d j o • j • j j j iy l k b l j a j C j k s j G C j k
21. e a ui g • L 1 2 fa • 1 1 p c 1 1 1 • 1 1 D n • ( ) ( • e 1 og 1L 1 S m
22.   
    
    
      
        
     sm Csm s LI lknt t ( ) ( ( y ) sa p l o 8 l s l l 8 R h y D S
23. 3 2 3 2
24. • / • E D • IC • L
25. L I ) 3 2L 5 E r C reh l C jc L • ro_m C • ( r in p C • 0:: 9 /. 1: :/.0 .6 2 / 9 ) (- ) 0: 3 reh P 1/ V jc
26.     
          
    
27. /2 ( & (& &( ) 3 2 • steps.{step-name}.outputs.parameters • steps.{step-name}.outputs.artifacts
28. 8 4 8 4 1 2 4 / 8 4 (  &*    &* &* ')"&* &*')$# &* 
      %  ! • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 ( 4 :8 ) .9 2 • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 4 :8 .9 2
29. 2 /4        
     
30. T 3E $ # !   "
          / 0 P S P 4
31. C G 3 1 / 4 RC A F E O . PI : AWSGCSMinio
     : : / .
32. 3 2 W / 1 2 W / 1 /  
         % %
33. 3 2/ 2 A B /
34. 4 / H S H B A R E R / 3
35. AC ( 3 3 ( ( ) 5 / 2 1 5 (
36. 6 / 2 2 3 • DC MO C z AC CK • D g mbR V rP D BK • Snv S S vR C E DC DC DCBK • A DB K • lI kIS R & C BK • D A C B gc b f b SYhI uRMH / 2 • C f b RL st D A C B i IRo Pp • DBC G R , S x • i I SaekY • p S x • /
37. 3 . • )/ • ( )/ • )/ • )/ • )/
38. 3 )3 138 (/ ( 2 /32 • ( 2 /32 )3 138 A • ( ( ES -1 g t w t • l AdeE ) yf yf A Rn T kpAES • o DP ES uBr h uBr h uBr h h h h
39. 3 3 / 39 2 T / 3 A 3 W P ( ( )) 2 T
40. E /0 G P - / Ec A • / -/ E T a C A PE S • 00 4 / 4 4 / / /- E UG TC
41. / 3 1 / 3 1 4 / 1 A
42. / 3 / 3 4 2 / A • • S G G • C M G
43. 3 4 4 • h E A N • /. GH C H 3. T c ge ./ 3 /. i /. a • / 4 i • . 4 /.
44. 21 /1 G G A io s h e W g h ea ei P b h m rlE d g k nu P bg h E p ht argo submit -f ci/hello.yaml git clone spec: templates: - name: approve suspend: {} 2 4
45. 2 / / n / D w P p t / ID C r a c e o d d / Ck d i hm / Csp DC Cn - name: build sidecars: - name: dind image: docker:18-dind securityContext: privileged: true mirrorVolumeMounts: true container: image: docker:18 env: - name: DOCKER_HOST value: 127.0.0.1 command: - sh - -c args: [" until docker ps; do sleep 3; done; ¥ docker build -t example . && ¥ docker save example > example.tar "] I 4 C 5 DA
46. 1 26 D C SO e D f g d PR R ia Be cCe A C i a Be Argo CD Sealed Secrets Controller Secrets Sealed Secrets Custom Resource Kubernetes API Server apply sync D 4 1241 / 24 : 4 / A 1 C A
47. / cg S A l n 2 . S 2 RFR voS t production develop spec: destination: server: https://kubernetes.default.svc source: targetRevision: master spec: destination: server: https://kubernetes.default.svc source: targetRevision: develop kunm Svo a e bgSrh P 2 1 py KJSr 2 dd o si o apiVersion: batch/v1 kind: Job metadata: generateName: post-sync- annotations: argocd.argoproj.io/hook: PostSync 47 7 4 2 4
48. 5 4 .
49. A 9 / / I 4/ /4 1 31
50. w s A / i g l / 5 32 i l l / 5 32 / / e t
    
    • 5 32 r vo k n • l e • TMG c E M gd G E c G MiN M M Ma fk 5 2 / 0t
    
    • E E / 5 32 u fk o r • TMG M e TA E / 5 32 p o
51. u Wo P 5 hEe la / 9 : C l iDat v R P PESE • kD cjD m W Ps SE • - • • / 9 : C.3 : 3 T P r S PE • 57 A1 2 5 5 7 A3 • 57 A1 2 5 5 A:: kD cjD bhgD W s E p . W :A 3f la bhgD E E w W s SE
52. 1 0 r @ A i ? 7
    
    
    e d o h Q      
     ng 0
53.  
    
54. 4 + 5 55 4 RS g u c g 4 o CTA T apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-project spec: description: Example Project sourceRepos: - '*' destinations: - namespace: guestbook server: https://kubernetes.default.svc clusterResourceWhitelist: - group: '' kind: Namespace namespaceResourceBlacklist: - group: '' kind: ResourceQuota 4 i P 4 mCTA R G p P tl m 44 g h s BG T D 44 4 5 O e r j apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd labels: app.kubernetes.io/name: argocd-cm data: dex.config: | connectors: - type: oidc id: google name: Google config: {} helm.repositories: | - url: https://kubernetes- charts.storage.googleapis.com name: sealed-secrets