apidays Australia 2023 - Building On-Premise Hybrid API Platforms, David Freeman, Sonrai

apidays Australia 2023 - Platforms, Products, and People: The Power of APIs
October 11 & 12, 2023
https://www.apidays.global/australia/

Building On-Premise Hybrid API Platforms
David Freeman, Founder of Sonrai

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. 1 01 1 1 0 01 1 0 1 1 0 1 1 1 0 01 1 0 1 1 1 0 01 1 1 0 1 0 01 1 0 00 01 01 1 0 0 0 1 1 1 01 1 1 0 01 1 0 1 1 0 1 1 1 0 01 1 0 1 1 1 0 01 1 1 0 1 0 01 1 0 00 01 01 1 0 0 0 1 1 Building an On-Premise Hybrid API Platform Sonrai
3. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 • Architects • Lead engineers • Technical project managers Who is this for?
4. Getting a few things out of the way Before we talk about the delivery of an On-Premise API Platform
5. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Why would you build on-premise when there’s so much cloud? …it doesn’t make any sense Getting a few things out of the way Regulatory Location
6. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 ~100ms
7. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Why would you build on-premise when there’s so much cloud? …it doesn’t make any sense Getting a few things out of the way Regulatory Resiliency Location
8. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0
9. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Why would you build on-premise when there’s so much cloud? …it doesn’t make any sense Getting a few things out of the way Regulatory Political Resiliency Location
10. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Never underestimate the weight of the enterprise Getting a few things out of the way Start Launch Launch Start Small Business Enterprise Start Launch Me at home
11. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 “organizations design systems that mirror their own communication structure” Conway’s Law Network API Platform OpenShift / Infrastructure APIs Networks team Application team(s) Integration team Infrastructure team Systems / Technology Stack Organisation Project manager Architect Engineering Lead Product Owner Platform Owner
12. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 • Approval cycles – e.g. budget, design, change requests, ops readiness • Testing co-ordination – e.g. PEN test engagement • Technology readiness – e.g. infrastructure • Design pivots during build – e.g. new environments • Inherited limitations – e.g. network, storage and other technology decisions Other enterprise challenges
13. Our On-Premise API Platform Experience The ins and outs, tips and tricks, from our build experience
14. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 • Build an on-premise API Platform • Extension to existing API Platform on the Cloud (Hybrid) • Must run active-active on 2 sites in 2 states • Must survive total cloud outage • Must be up ALL the time, yes ALL the time, like 99.999+% • Everything On-premise capable – image registry, metrics, logging, code repo, etc. What was the problem statement?
15. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Timeline to delivery Design Build Test Operate July August September October 5 environments on 4 clusters in 2 sites Highly-available 2-site on-premise API Platform Performance, resiliency and penetration testing 99.999+% availability Launch • 5-6 month project • 2 x people (cross-skilled) • Architect/designer also builds • Sandpit environment • Builders provide support
16. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Solution overview API Client applications Private Data Centres - SYD/QLD POP Router Internet Load Balancers Firewalls API Mgmt Azure Cloud Load Balancer s Virtual LB Firewalls Azure DevOps Azure Cloud Azure SE Azure E Internal Firewalls Secure Backend Identity/Logging Mission critical API connections Lower tier API connections OpenShift Clusters APIM Management Plane connectivity On-Premise Backend Systems APIM self-hosted gateway Virtual LB Azure to DC Private Network public.api.com.au partner.api.com.au DevOps and Pipelines Internet-facing gateway for mission-critical APIs Mgmt plane connectivity for API updates, heartbeat, telemetry Azure DevOps for YAML manifests, config, etc Existing internet- facing APIs route to Azure cloud API gateway © Copyright 2023, Sonrai Consulting Pty L
17. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Physical infrastructure Sydney DC Brisbane DC Dev Environment VMware OpenShift Cluster Master master1 master2 master3 Nodes (machine-set) node1 node2 node3 Production Environment Physical server environnent Dell Dell Dell OpenShift Cluster Master master1 master2 master3 Nodes (machine-set) node1 node2 node3 API-GWY-PROD-SYDNEY Production Environment Physical server environnent Dell Dell Dell OpenShift Cluster Master master1 master2 master3 Nodes (machine-set) node1 node2 node3 API-GWY-PROD-BRISBANE Non-Prod Environment VMware OpenShift Cluster Master master1 master2 master3 Nodes (machine-set) node1 node2 node3 API-GWY-PREPROD-SYDNEY node4 API-GWY-SIT-SYDNEY • Physical infra delivery time • Data centre space • Network capacity • Matchy-matchy © Copyright 2023, Sonrai Consulting Pty Ltd
18. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Kubernetes deployment Sydney DC Brisbane DC Dev Environment OpenShift Cluster Non-Production Environment OpenShift Cluster Namespace apim-dev API-GWY-DEV-SYDNEY pod1 pod2 etc.. Namespace apim-sit API-GWY-SIT-SYDNEY pod1 pod2 etc.. Namespace apim-ppd API-GWY-PREPROD-SYDNEY pod1 pod2 etc.. Namespace apim-sit API-GWY-SIT-SYDNEY pod1 pod2 etc.. Production Environment OpenShift Cluster Namespace apim-prd API-GWY-PROD-SYDNEY pod1 pod2 etc.. Production Environment OpenShift Cluster Namespace apim-prd API-GWY-PROD-BRISBANE pod1 pod2 etc.. 1 2 3 4 • Config / Code drift (CICD) • Deploy (in Dev) as you design • Bring security on the journey • See later for sample deployment manifest © Copyright 2023, Sonrai Consulting Pty L
19. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Internal Network Internal Network Backend Systems PPD PRD Client applications External Gateway Network PPD Aplication PRD Application External F5s External Firewalls Internet Internet Internal F5s Internal Firewalls Internal systems api-prd.local.url -> 192.prd.prd.99:443/ 31884 Load Balancing Healthcheck: Endpoint: /status-0123456789abcdef Host : api-gwy-prd.local.url IP : 10.syd.prd.17-19:30884, 10.brs.prd.17-19:30884 Internal Firewalls Internal Firewalls Sydney DC - SYD PPD-OPENSHIFT : VLANPPD : 10.syd.ppd.0/24 OpenShift Cluster Cluster interfaces 10.syd.ppd.17 - 19 Namespace apim-ppd Deployment APIM Self-Hosted Gateway Pre-Production serviceNetwork – 172.ppd.ppd.0/16 API-PPD-APIM-SYD-live-traffic 172.ppd.n.n Workload replicaSet - 172.ppd2.0.0/16 API-PPD-APIM-SYD-[GUID] 172.ppd2.n.n Namespace openshift-ingress OpenShift Route api-ppd-apim-route Location : https://api-ppd.local.url Egress IP - PPD 10.syd.ppd.67 10.syd.ppd.68 10.syd.ppd.69 Ingress-controller ingress-apim-ppd Ports: 80/30773, 443/31773, 1936/30922 Sydney DC - SYD PRD-OPENSHIFT : VLANPRD : 10.syd.ppd.0/24 OpenShift Cluster Cluster interfaces 10.syd.prd.17 - 19 Namespace apim-prd Deployment APIM Self-Hosted Gateway Production serviceNetwork – 172.prd.prd.0/16 API-PRD-APIM-SYD-live-traffic 172.prd.n.n Workload replicaSet - 172.prd2.0.0/16 API-PRD-APIM-SYD-[GUID] 172.prd2.n.n Namespace openshift-ingress OpenShift Route api-prd-apim-route Location : https://api-prd.local.url Egress IP - PRD 10.syd.prd.77 10.syd.prd.78 10.syd.prd.79 Ingress-controller ingress-apim-prd Ports: 80/30884, 443/31884, 1936/33590 Sydney DC - DEV-OPENSHIFT : VLANdev : 10.dev.dev.0/24 OpenShift Cluster Cluster interfaces 10.dev.dev.160 -163 Namespace apim-dev Deployment APIM Self-Hosted Gateway Dev serviceNetwork – 172.dev.dev.0/16 API-DEV-APIM-SYD-live-traffic 172.dev.n.n Workload replicaSet - 172.dev2.0.0/16 API-DEV-APIM-SYD-[GUID] 172.dev2.n.n Namespace openshift-ingress OpenShift Route api-dev-apim-route Location : https://api-dev.local.url Egress IP - DEV 10.dev.dev.138 10.dev.dev.139 Ingress-controller ingress-apim-dev Ports: 80/30338, 443/31338, 1936/30922 api-ppd.local.url -> 192.ppd.ppd.98:443/ 31773 Load Balancing Healthcheck: Endpoint: /status-0123456789abcdef Host : api-gwy-ppd.local.url IP : 10.syd.ppd.17-19:30773, 10.brs.ppd.17-19:30773 partner.api.com.au partner.preprod.api.com.au Network • External access © Copyright 2023, Sonrai Consulting Pty Ltd
20. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 External Gateway Network External F5s External Firewalls Internet api-prd.local.url -> 192.prd.prd.99:443/ 31884 Load Balancing Healthcheck: Endpoint: /status-0123456789abcdef Host : api-gwy-prd.local.url IP : 10.syd.prd.17-19:30884, 10.brs.prd.17-19:30884 Internal Firewalls Sydney DC - SYD PPD-OPENSHIFT : VLANPPD : 10.syd.ppd.0/24 OpenShift Cluster Namespace apim-ppd Deployment Namespace openshift-ingress OpenShift Route api-ppd-apim-route Location : https://api-ppd.local.url Ingress-controller ingress-apim-ppd Ports: 80/30773, 443/31773, 1936/30922 Sydney DC - SYD PRD-OPENSHIFT : VLANPRD : 10.syd.ppd.0/24 OpenShift Cluster Namespace apim-prd Deployment Namespace openshift-ingress OpenShift Route api-prd-apim-route Location : https://api-prd.local.url Ingress-controller ingress-apim-prd Ports: 80/30884, 443/31884, 1936/33590 /30922 api-ppd.local.url -> 192.ppd.ppd.98:443/ 31773 Load Balancing Healthcheck: Endpoint: /status-0123456789abcdef Host : api-gwy-ppd.local.url IP : 10.syd.ppd.17-19:30773, 10.brs.ppd.17-19:30773 partner.api.com.au partner.preprod.api.com.au Network Load-balancing across Syd and Bris via healthcheck – “statusendpoint” Specific port for ensuring traffic routes to a specific ingress- controller Ingress-controller port 31884 OCP Route (or ingress in K8) to route the traffic to the right app (APIM) based on URL in the request • Load-balancing across nodes and clusters • Ingress-sharding • Careful of SNI tagging • Routing traffic complexity • External access © Copyright 2023, Sonrai Consulting Pty L
21. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Security • Network • Ingress-sharding • Network policies • Egress IPs per namespace • Firewall rules • See IBM red-paper 5641* * By Loïc Fura, Joyce Mak, Veera Santoshi External F5 PRD API Ingress VIP Physical network VLAN DEFAULT Brisbane OpenShift Cluster Sydney OpenShift Cluster OpenShift Node 10.syd.prd.17 OpenShift Node 10.syd.prd.18 OpenShift Node 10.syd.prd.19 Namespace: apim-prd OpenShift Node 10.brs.prd.17 OpenShift Node 10.brs.prd.18 OpenShift Node 10.brs.prd.19 Namespace: apim-ppd api-prd-syd.local.url API-PRD api-prd-bris.local.url API-PRD api-ppd-bris.local.url API-GWY-PPD Namespace: openshift-ingress OpenShift Service Network OpenShift Cluster Network PRD ingress PPD ingress Default ingress VLAN PRD VLAN PPD EgressIP A (Prod) EgressIP B (Ipt) POD Service Ingress Controller Sydney OpenShift Environment nodeport nodeport Sydney OpenShift Default Load Balancer Internal LTM Sydney OpenShift PPD Load Balancer Internal LTM Sydney OpenShift Prod Load Balancer Internal LTM On-Premise API PRD Gateway VIP Internal LTM api-prd.local.url On-Premise API PPD Gateway VIP Internal LTM api-ppd.local.url External LTM partner.api.com.au Extneral F5 PPD API Ingress VIP External LTM partner.preprod.api.com.au Partner2 F5 PRD API Ingress VIP External LTM Partner2.api.net.au External F5 Environment Internal F5 Environment Brisbane OpenShift Default Load Balancer Internal LTM Brisbane OpenShift PPD Load Balancer Internal LTM Brisbane OpenShift Prod Load Balancer Internal LTM Brisbane OpenShift Environment api-ppd-syd.local.url API-PPD PPD ingress PRD ingress Default ingress Namespace: apim-prd Namespace: apim-ppd Namespace: openshift-ingress © Copyright 2023, Sonrai Consulting Pty Ltd
22. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Namespace: apim-ppd Namespace: apim-prd partner.api.prod.com.au partner.api.preprod.com.au Production backend systems Pre-production backend systems OpenShift Environment api-prd.local.url API-PRD api-ppd.local.url API-PPD Connect Bound Bound Connect Connect EgressIP A (Prod) EgressIP B (Preprod) API-PRD only API-PPD only PRD ingress Port: 31abc PPD ingress Port: 31xyz Security • Network F5 routes traffic attempting to hit particular URLs to unique ports. Ingress controllers route traffic to applications in particular namespace Network policies prevent applications from cross- communicating Dedicated EgressIPs per namespace allow outbound firewall segregation of traffic © Copyright 2023, Sonrai Consulting Pty Ltd
23. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 • Local metrics (and cloud metrics) Operations • Both Local and Cloud-metrics • TSDB on PVC • Set resource limits © Copyright 2023, Sonrai Consulting Pty Ltd OpenShift Cluster Namespace apim-metrics Namespace apim-ppd Deployment APIM Self-Hosted Gateway Pre-Production serviceNetwork – 172.ppd.ppd.0/16 API-PPD-APIM-SYD-live-traffic 172.ppd.n.n Workload replicaSet - 172.ppd2.ppd2.0/16 API-PPD-APIM-SYD-[GUID] 172.ppd2.n.n metrics- pvc serviceNetwork – 172.ppd.ppd.0/16 apim-metrics-prometheus Type: LoadBalancer 172.ppd.ppd.nnn:9090 replicaSet - 172.ppd2.ppd2.0/16 apim-metrics-prometheus 172.ppd2.n.n apim- metrics- config serviceNetwork – 172.ppd.ppd.0/16 apim-metrics-statsd Type: NodePort 172.ppd.ppd.nnn:8125 apim-metrics-statsd 172.ppd2.n.n OpenShift Route api-ppd Location : https://[service-name]-[openshift- infra-name].local.url Prometheus Database Metrics output
24. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 • Local logging (using filebeat) Operations • Privileged permissions • Indexing, dissecting and parsing • ClusterLogging vs Filebeat OpenShift Cluster PRD Sydney On-premise ELK instance Logstash node3 node2 node1 Namespace: apim-filebeat DaemonSet: filebeat DaemonSet: filebeat DaemonSet: filebeat filebeat -sa filebeat -sa filebeat -sa Egress IP – PRD Sydney 10.syd.prd.17-19 Internal Firewalls Port: 5044 /var/log/containers/ {{container.id}}.log /var/log/containers/ {{container.id}}.log /var/log/containers/ {{container.id}}.log filebeat Config Map © Copyright 2023, Sonrai Consulting Pty Ltd
25. Pilot traffic Go live traffic Go live OnPrem Site1 OnPrem Site2 32 x 5xx errors from 500,000+ calls to-da (99.996% success rate)
26. 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 Download: • Presentation, • Full example detailed design, • Visio diagrams, • Example YAML manifests
27. GET /your-api-program {on-track} GET /your-api-program {on-track} Sonrai.com.au