Cloud Native Components in VTT's Data Pipelines

Transcript

1. Cloud Native Components in VTT's Data Pipelines 3.12.2019 VTT – beyond the obvious 1
2. What is VTT? § Technical Research Centre of Finland § About 2000 researchers § Wide array of topics • Nuclear safety • Printed electronics • Food science • Data-driven services § About 200 projects develop software yearly, involving 5% of personnel § Yearly 10-20 projects have to gather new datasets for research 3.12.2019 VTT – beyond the obvious 2
3. 3.12.2019 VTT – beyond the obvious 3 Motivation
4. A standard base setup § Project team must own the deployment(s) § Empowering the researchers • They are experts of their fields (e.g.: machine learning) • The best way to use their talent is to do research § We want to give them tools that takes care of the basics § Helps with GDPR compliance § Customizable 3.12.2019 VTT – beyond the obvious 4
5. 3.12.2019 VTT – beyond the obvious 5 Data AI / ML The researchers want …
6. 3.12.2019 VTT – beyond the obvious 6 Kubernetes Easy deployment, secure communication, monitoring Ingress with TLS Auth Anonymization …and what is deployed. Data AI / ML
7. 3.12.2019 VTT – beyond the obvious 7 Pieces to the (cluster) puzzle
8. RPC protocol § Http/2 based RPC protocol § Protobuf based data-object / service definition § Client / server bindings are generated § Many target languages § Effective, binary data-representation § gRPC-web brings support for web- clients 3.12.2019 VTT – beyond the obvious 8
9. Ingress § When using gRPC a LoadBalancer type Service is not ideal • Layer 4 vs Layer 7 § Takes care of TLS termination § We had previous experience with Envoy, but other options are also available (e.g.: Nginx, Traefik) § All of them offer features beyond Ingress specification 3.12.2019 VTT – beyond the obvious 9
10. TLS certificate management § Certificates from Let’s Encrypt § Cert-manager by Jetstack • Supports HTTP and DNS based validation • HTTP validation works only if Ingress objects work • Only DNS based validation supports wildcard domain names 3.12.2019 VTT – beyond the obvious 10
11. Service mesh § Original goals: • Monitoring with no change to service code • Pre-configured dashboard • Lightweight (memory, CPU) § mTLS originally seen as nice extra • With certain data types (sensitive personal information, e.g.: health data) it helps a lot with GDPR compliance • Some performance penalty § Nice functions we don’t utilize much yet • Retry budget, traffic shifting 3.12.2019 VTT – beyond the obvious 11
12. Automated provisioning § Infrastructure as code, using real programming languages • JavaScript / TypeScript (Node.js) • Python § Automatic and manual dependency § Great Kubernetes support • Programmatic Kubernetes objects • Helm charts / Standalone Yaml files • Waiting for components to became ready 3.12.2019 VTT – beyond the obvious 12
13. GitOps § Researchers can easily deploy modifications with a tool they already know (Git) § Flux for the actual GitOps deployment § Bitnami Sealed Secrets so secrets can be checked-in to Git § Weave Flagger for canary deployments (not yet used in production) 3.12.2019 VTT – beyond the obvious 13
14. 3.12.2019 VTT – beyond the obvious 14 Assembly
15. 3.12.2019 VTT – beyond the obvious 15 Create cluster Install service-mesh Install cert-manager Install ingress-controller Register DNS name Request certificate Install GitOps tools Serve application
16. 3.12.2019 VTT – beyond the obvious 16 Demo time!
17. Presenter info § Zsolt Homorodi, Senior Specialist, VTT § @HaZseTata § https://github.com/hazsetata § https://gitlab.com/hazsetata § Demo: https://gitlab.com/hazsetata/kubedemo 3.12.2019 VTT – beyond the obvious 17