Moving to Microservices @vex

Moving from a monolith to micro-services. Desk supporting the story told on the GDG Cloud Belgium meetup on June 26th, 2018

Transcript

1. Shop smart, feel great Moving to Microservices @vex
2. Big Data/Cloud Architect @ vente-exclusive.com Google Developer Expert - follow @alexvb Alex Van Boxel
3. Lead Architect @ vente-exclusive.com follow @dot_i Nico Francois
4. The views and opinions expressed in this talk are those of the authors and do not necessarily reflect the official position of Vente-Exclusive or Google
5. The Monolith Production database Reporting database BigQuery Cloud Pub/Sub Nightly table transfers Cloud Storage Channel interactions (CAL events) .Net Applications Tableau Server Microsoft Excel Cloud Dataflow
6. Why?
7. Platform Snowflake vs Phoenix
9. Platform
10. Cloud Google Edge POP Cloud Load Balancing Cloud CDN Cloud Firewall Rules Cloud Armor Google Edge POP Cloud Armor Google Edge POP Cloud Armor Cloud DNS Identity-Aware Proxy Google Edge POP Cloud Armor europe-west1 zone-b Kubernetes Engine zone-c Kubernetes Engine zone-d Kubernetes Engine europe-west2/3/4 (future) zone-x Kubernetes Engine zone-y Kubernetes Engine zone-z Kubernetes Engine Storage Images
11. Single Responsibility i18n auth
12. Continuous Integration & Continuous Delivery
13. Everything Scripted
14. CLI
15. Proxied Environments staging auth pay i18n cart
16. Proxied Environments staging feature-foo auth pay i18n cart cart
17. Proxied Environments staging feature-foo auth pay i18n cart auth pay i18n cart
18. Proxied Environments staging feature-foo feature-bar auth pay i18n cart auth pay i18n cart auth pay i18n cart
19. Local Proxy staging local auth pay i18n cart auth pay i18n cart
20. Demo Time
21. And stick with them Making hard choices
22. Contracts and Conventions boreporting-api
23. Contracts and Conventions boreporting-api
24. Contracts and Conventions tableauauth-api
25. Technology choices Cloud Pub/Sub
26. Technology choices Cloud Pub/Sub
27. Contracts and Conventions But make sure you allow for growth
28. And stick with them Living on the ship
29. Scalability
30. Fault Isolation
31. Operations and SRE
32. Operations and SRE
33. Observability Kubernetes cluster #FCE4EC pod #E8F5E9 Stackdriver Cloud Load Balancing Google Edge POP Cloud Armor Cloud Endpoints Security, Trace, Monitoring, Rate limit Cloud Debugger Monitoring Logging Error Reporting Trace Debugger Forseti Security Microservice
34. Operations and SRE SRE
35. Demo Time
36. Road ahead The Future
37. Future Product Support
38. Future Service Mesh
39. Future The Road To GitOps
40. Future PreGitOps
41. Future Kubernetes Operators
42. Declarative apiVersion: pubsub.quantum.build/v1 kind: PubSub metadata: name: ordermanagement-api spec: module: ordermanagement topics: - name: return-announced - name: return-created - name: return-completed - name: cancellation-completed - name: cancellation-confirmed - name: cancellation-created subscriptions: - topic: carts.cart-confirmed - topic: logisticalmanagement.cancellation-processed - topic: logisticalmanagement.shipment-shipped - topic: logisticalmanagement.shipment-delivered - topic: logisticalmanagement.return-processed
43. Pushing it to k8s > kubectl apply -n=crd -f artifacts/example-ordermanagement-crd.yaml pubsub "ordermanagement-api" created > kubectl get events -n=crd pubsub "ordermanagement-api" created > kubectl get events -n=crd LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE 1m 1m 1 ordermanagement-api.1534a380fdb5a113 PubSub Normal TopicCreated pubsub-operator Topic crd.ordermanagement.return-announced created 1m 1m 1 ordermanagement-api.1534a3813617570d PubSub Normal TopicCreated pubsub-operator Topic crd.ordermanagement.return-created created 1m 1m 1 ordermanagement-api.1534a381ce6c0f99 PubSub Normal TopicCreated pubsub-operator Topic crd.ordermanagement.return-completed created 1m 1m 1 ordermanagement-api.1534a382749e3d58 PubSub Normal TopicCreated pubsub-operator Topic crd.ordermanagement.cancellation-completed created
44. Kubernetes Operator CR Operator k8s
45. Future PubSub Google Service Accounts Ingress and Load Balancer Proxied Environments and DNS
46. Future Could improve more: the Deployment
47. Deployment example (meta data) apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: labels: version: 1.7.3-develop_d61ebc6 name: auth-api namespace: develop
48. Deployment example (1/9) spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 2 selector: matchLabels: component: api module: auth strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: component: api module: auth version: 1.7.3-develop_d61ebc6 spec:
49. Deployment example (2/9) affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: component: api module: auth topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: false containers: - env: - name: GOOGLE_PROJECT_ID value: quantum-cloud-test - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name
50. Deployment example (3/9) - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: GOOGLE_APPLICATION_CREDENTIALS value: /var/run/secrets/quantum/google-service-account/service-account.json - name: PORT value: "5000" - name: MongoDB__Database value: develop-auth - name: ASPNETCORE_ENVIRONMENT value: Test - name: ASPNETCORE_URLS value: http://*:5000 - name: STATIC_CONTENT_URL value: https://test-static.quantum.build/auth/ 1.7.3-develop+d61ebc6/
51. Deployment example (4/9) - name: PUBLIC_CONTENT_URL value: https://test-content.quantum.build/ - name: SECRETS_SHARED value: /var/run/secrets/quantum/shared - name: SECRETS value: /var/run/secrets/quantum/auth-api image: eu.gcr.io/quantum-cloud-test/ auth-api:1.7.3-develop_d61ebc6 imagePullPolicy: IfNotPresent name: auth-api ports: - containerPort: 5000 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /probes/readiness port: 5000 scheme: HTTP initialDelaySeconds: 6 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1
52. Deployment example (5/9) readinessProbe: failureThreshold: 3 httpGet: path: /probes/readiness port: 5000 scheme: HTTP initialDelaySeconds: 6 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1 resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/ssl/certs name: ssl-certs - mountPath: /var/run/secrets/quantum/google-service-account name: google-service-account - mountPath: /var/run/secrets/quantum/shared name: secrets-shared - mountPath: /var/run/secrets/quantum/auth-api name: secrets-auth-api
53. Deployment example (6/9) - args: - -s - $(SERVICE_NAME) - -v - $(SERVICE_CONFIG_ID) - -k - /var/run/secrets/quantum/google-service-account/service-account.json - -z - probes/readiness - -p - "5001" - -a - 127.0.0.1:5000 env: - name: SERVICE_NAME value: auth.endpoints.quantum-cloud-test.cloud.goog - name: SERVICE_CONFIG_ID value: 2018-05-29r1 image: gcr.io/endpoints-release/endpoints-runtime:1.13.0 imagePullPolicy: IfNotPresent name: auth-proxy
54. Deployment example (7/9) ports: - containerPort: 5001 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /probes/readiness port: 5001 scheme: HTTP initialDelaySeconds: 15 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "1" memory: 75Mi requests: cpu: 25m memory: 30Mi
55. Deployment example (8/9) terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/quantum/google-service-account name: endpoint-service-account dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - hostPath: path: /etc/ssl/certs type: Directory name: ssl-certs - name: google-service-account secret: defaultMode: 420 secretName: gsa-qm-auth-api - name: endpoint-service-account secret: defaultMode: 420 secretName: gsa-qm-endpoint-proxy
56. Deployment example (9/9) - name: secrets-shared secret: defaultMode: 420 secretName: shared - name: secrets-auth-api secret: defaultMode: 420 secretName: auth-api
57. But what's unique? Almost all our platform deployments are the same What if it could be simpler?
58. Deployment example (all of it!) It could boil down to this apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deploy.quantum.build/product: "photon" deploy.quantum.build/endpoint: "2018-05-29r1" deploy.quantum.build/useMongo: true deploy.quantum.build/useGrpcTranscoding: false deploy.quantum.build/funckyNewFeature: true labels: version: 1.7.3-develop_d61ebc6 module: auth component: api name: auth-api namespace: develop
59. How does it work Admission Hooks
60. Admission Hook Deployment Deployment k8s Your hook here
61. Future GitOps Push or Pull
62. Future What Else?!
63. Future Better Diagnostics
64. Future
65. Future Declarative Dashboards
66. Future Declarative Alerting
67. People Other Cool Stackdriver Stuff
68. People
69. People Elephant First
70. People Contract First
71. starting over, make it count Opportunities
72. Opportunities Streaming operational data to the data lake
73. Opportunities Angular Front-end MongoDB Databases Angular Back-Office Microservices .Net Event sourcing Cloud Pub/Sub Cloud Bigtable Client events -- clickstream (sessions) -- interesting business events DATA microservices Python / GO DATA microservices Python / GO SparkPost ADYEN External events Cloud Pub/Sub Payments E-mail MS SQL IMS, Area51 Entity builder + KPI calculation DataFlow streaming BigQuery Events processing Dataflow streaming Production data -- not accessible outside IT -- splitted per microservice Reporting data -- real-time! -- full change history -- self-service BI -- excel report generation -- permission based -- GDPR! -- SaaS -- Standard SQL language -- High-availability -- low-latency -- real-time -- PaaS -- backup + history Elasticsearch elastic.io managed -- search (backoffice only atm) -- future: site search Website backoffice PHOTON
74. Opportunities Data Lake
75. Opportunities Data Lake enables Ops
76. Opportunities Data Lake for GDPR compliance Forseti Security
77. Opportunities Make it Count
78. Shop smart, feel great THANK YOU twitter @alexvb @dot_i