A Whirlwind Tour of Infra

A Whirlwind Tour of Infra

In this talk we cover how technology shapes architecture, and new developments in technology advance architecture.


Michael Hamrah

January 22, 2019


  1. A Whirlwind Tour of Infra Or A Brief History of

    Cloud Or Why Everything Is Hard 1
  2. A Story of Binaries and Processes 2

  3. A binary is just data on disk. A set of

    instructions. A process is a binary loaded into memory with a pid (loaded from another process!) CPUs execute instructions, which use memory and IO via syscalls 3
  4. ELF and PE see https://news.ycombinator.com/item?id=8029564 4 ❯ ldd dotnet linux-vdso.so.1

    (0x00007ffe2979b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5b5e4d0000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5b5e4a libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5b5e libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b5e198000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b5e17e000 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b5df94000) /lib64/ld-linux-x86-64.so.2 (0x00007f5b5e4f5000) Links are just other files loaded into memory!
  5. A scripting language starts with a binary…. That loads instructions

    and interprets/optimizes/executes high-level instructions myscript.sh node index.js ruby app.rb 5
  6. Sometime scripts are compiled to an intermediate language (as a

    library) and interpreted/optimized/executed... dotnet myapp.dll java myapp.jar wasm myapp.wasm Still requires linked libraries! 6
  7. Sometimes you can have just a binary… Statically compiled… With

    dependencies… CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags '-w -extldflags "-static"' -o mybin *.go 7
  8. In The Beginning • Hardware Ruled The World • Infrastructure

    Was Static • Processes Were bad • Vendors Were Dictators • Things Took Months/Years 8
  9. 9 F5 IIS APACHE DB SAN/RAID Apache with: mod_perl mod_php

    phusion passenger IIS Java (+Apache) with: JBoss Websphere Tomcat VMWare Public IP Private DNS
  10. Then Came Google (and AWS) 10 And “We’re not Google

  11. Hardware -> Software 11 HAProxy over F5 Hadoop Distributed Data

    Nginx over Apache (light over heavy)
  12. Servers Were Ephemeral Like clouds. Dotting a blue sky. Moving

    and morphing with the wind. 12
  13. 13 But there were still problems • Did not solve

    shared libraries • Did not solve configuration • Routing had limitations • Did not solve deploy
  14. 14 Immutable Infrastructure My first public talk! • Use Packer

    • Bake Images • Launch with automation • CFN, ASGs, ELBs, R53, S3, EBS • Jenkins, a UI for bash
  15. 15 And all was good in the world. But darkness

    lurked... • Things were slow • Hard to debug • Bespoke automation
  16. 16 And Then There Was Docker Which is just a

    binary. With its linked libraries. Bundled in a tarball. With better automation. And easier process management.
  17. 17 Containers Everywhere Architecture rethought. 3 tier, fat process ->

    Microservices Back to square one… CoreOS Mesosphere Kubernetes Docker Swarm Jaeger Istio... Lots of processes Lots of ad-hoc automation
  18. 18 Requirements • Run Code How how how?!?!?!!?

  19. 19 Requirements • We need servers (maybe not?!?!) ◦ We

    need to run processes • We need to configure these things • We need to install binaries on these ‘things’ • We need to configure these binaries • We need to run these binaries as processes • We need to secure and manage these things and processes • We need to route traffic to these processes • We need to update these things • We need to update these processes How how how?!?!?!!?
  20. 20 Observability Continuous Integration Continuous Delivery/Deployment Runtime (Kubernetes/Docker/App Servers) Server

    Infrastructure (AWS) Operations Configuration Management Environment Management Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs Security
  21. There is no right answer, only various degrees of wrong.

    We experiment, learn, decide, act, rinse, repeat and improve!
  22. 22 SignalFx, Logz.io, New Relic Jenkins Spinnaker/Octopus Kubernetes + Istio

    AWS Spinnaker, kubectl, Kubernetes, ad-hoc Kubernetes/Octopus Terraform Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs IAM, RBAC, Networking, VPN, Secrets
  23. 23 Welcome to Namely Infra AWS (Virginia) Production Int Stage VendorX Ops Portal IT An environment is: • An AWS account and permissions • A VPC • Route tables • Everything required to run Namely • The ability to deploy components
  24. Easy CIDR: A little lesson in IPs and

    An IP4 address is 32 bits That’s 32 1’s and 0’s 11110000 11110000 11110000 11110000 The /8 and /16 denotes how many bits ‘to keep’ (Big E) This denotes how IP’s are allocated A route table directs an IP range to a target
  25. 25 Environment Basics a Peering Public ELB(s) Internet Gateway Server1

    Server2 Jumpboxes Server3 VPC A bunch of RDS Alotta ElasticCache Some Aurora CloudFront S3 Kubernetes 15 workers 3 masters 5 etcd
  26. 26 Not so simple, though Zone 1a, private subnet, @ Zone 1c, private subnet, Zone 1d, private subnet, @ Node Node Node Node Node Node Node Node Node Zone 1a, public subnet, Zone 1c, public subnet, Zone 1d, public subnet, Public ELBs With Public IPs Private ELBs With Subnet IPs Try using the dig command to find out how dns names are mapped to IPs ENI ENI ENI SG SG SG SG SG SG SG SG SG
  27. 27 Automation #1: Terraform https://github.com/namely/tf2

  28. 28 Kubernetes • Allows us to run containers ◦ A

    container is just a binary, with its dependencies • Abstracts node-management issues for processes ◦ IP addresses, ports, security, quotas ◦ Also in this space: ECS, Docker-Swarm, GAE • Built-in config, secret, service discovery, scaling management • Automates some cloud infra, like load balancers • Foundational-focused API (doesn’t solve some things) ◦ Allows for extensibility
  29. 29 Kubernetes Cluster Etcd0 Etcd1 Etcd2 Etcd3 Etcd4 Master0 Master1

    Master2 Worker0 Worker2 Worker3 Worker1 Worker... Worker15 State is stored here Does most of the k8s work Where stuff runs
  30. 30 EKS Cluster Worker0 Worker2 Worker3 Worker1 Worker... Worker15 Where

    stuff runs Better networking
  31. 31 Kubernetes Pods • The smallest unit that runs on

    a cluster • Has a name and labels • Has one, unique private IP address • Has one or more containers • All containers in a pod share networking, storage ◦ Can “see” eachother on localhost Pod Name: slug-bcddbcd8-1sa3a Labels: • app: slug Containers: • Slug • image: namely/slug • ports: ◦ 50051 • env • Istio • A ‘Sidecar’ is simply another container in a pod, usually auto-injected
  32. 32 32 32 Kubernetes Networking Kubernetes creates a private address

    space for pods, and handles routing across nodes Worker0 Worker1 Worker2 Pod0 Pod1 Kubelet Pod0 Pod1 Kubelet Pod0 Pod1 Kubelet
  33. 33 33 Kubernetes ReplicaSets • ReplicaSets create Pods • They

    specify how many instances of a pod should be running • Represents desired state ◦ Kubernetes will try to get current state to desired state • You usually don’t deal with these though • But they are why Pods are re-created after you delete them! ReplicaSet Name: slug-bcddbcd8-1sa3a Labels: • app: slug Spec: • Replicas: 2 • Selector ◦ Labels • Template ◦ Same as Pod! Pod Pod
  34. 34 Kubernetes Deployments • Deployments Manage ReplicaSets • You usually

    deal with these ◦ Through Spinnaker! • Deployments wind down old ReplicaSets and scale up new ones. • They support various strategies for how things are updated. Deployment Name: slug Labels: • app: slug Spec: • Replicas: 4 • Strategy ◦ Rolling-update • Selector ◦ Labels • Template ◦ Same as Pod! RS-old RS-new
  35. Other ways to run Pods • CronJobs • DaemonSets •

    Jobs • kubectl run • kubectl exec lets you ‘get into’ a pod
  36. Kubernetes Services • The ‘default’ way pods discover pods (L4)

    • Uses an internal DNS service ◦ Currently CoreDNS • Also uses a private network (10.3/16) • Uses labels to match service names with pods • Has three types. ◦ ClusterIP (default) ◦ LoadBalancer ▪ Allows external traffic to flow to internal pods ▪ On AWS creates an ELB ◦ Don’t worry about type three Service Name: slug Labels: • app: slug Spec: • Type • Ports • Selector ◦ Labels
  37. Time for the OSI Stack!!!!!! Layer 7 Layer 5/6 Layer

    4 Layer 3 Layer 2 Layer 1 Application Blah Transport Network Data Link Physical HTTP/HTTP2 TCP/UDP IPv4/IPv6 Ethernet* Raw wires(less) This is important This is important
  38. Kubernetes Ingress • Services only speak L4-TCP/UDP ◦ ‘Limited’ routing

    (single service) • Ingress is L7. It knows about http and http2 • Not natively implemented by Kubernetes ◦ Only schema is defined ◦ Third parties implement ◦ We tried Contour, Nginx, and Istio • Allows us to compose, shape and route traffic declaratively ◦ Used for gRPC traffic ◦ Used for our APIs ◦ Used to better handle AATE egress Ingress Name: slug Labels: app: slug Spec: rules: - host: '*.i.namely.com' http: paths: - backend: serviceName: slug servicePort: 80 path: /api/slug
  39. All Together Now ELB/ILB Ingress Service Deployment Replica Set Pod

  40. Sadly not perfect • Load Balancing for gRPC • How

    do you determine ‘healthy’ pod • No standard metrics, tracing • No way to test failures • Can’t optimize traffic (inter-zone) • Standard retries, fail-fast
  41. Observability • Bypasses K8S Service ◦ No DNS • Speaks

    L4 and L7 • Emits stuff for us • Big plans for Namely ◦ A/B testing ◦ Failure testing ◦ Better traffic management Some Pod App Container Istio Other Pod App Container Istio Straight to Pod’s IP Hey K8S, tell me every service and every pod IP
  42. Is My Service Working?

  43. Is My Service Working?

  44. 44 Is My Service Working?

  45. We Must Automate If we are engineering processes and solutions

    that are not automatable, we continue having to staff humans to maintain the system. If we have to staff humans to do the work, we are feeding the machines with the blood, sweat, and tears of human beings. Think The Matrix with less special effects and more pissed off System Administrators engineers.
  46. Automation is about finding the right level of abstraction Want

    to do things easily Want to standardize… Yet allow for customization Build knowledge Find good tools and happy path
  47. Kubernetes CRDs Allows you to extend the native Kubernetes API

    We will use this for declarative management of Namely-things (like estuary -> spinnaker) Service creation, secrets, Pager Duty, Dashboards, Databases, Redis Apply yaml just like Deployments, Services, Ingress (Service Catalog) NamelyService Name: employee Spec: Team: hcm Requirements: - Redis - Postgres Observability: - Bugsnag - PagerDuty - SignalFx
  48. Everything is an Investment We want a return. We must

    build up on what we’ve done.