Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Whirlwind Tour of Infra

A Whirlwind Tour of Infra

In this talk we cover how technology shapes architecture, and new developments in technology advance architecture.

Michael Hamrah

January 22, 2019

More Decks by Michael Hamrah

Other Decks in Technology


  1. A Whirlwind Tour of Infra Or A Brief History of

    Cloud Or Why Everything Is Hard 1
  2. A binary is just data on disk. A set of

    instructions. A process is a binary loaded into memory with a pid (loaded from another process!) CPUs execute instructions, which use memory and IO via syscalls 3
  3. ELF and PE see https://news.ycombinator.com/item?id=8029564 4 ❯ ldd dotnet linux-vdso.so.1

    (0x00007ffe2979b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5b5e4d0000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5b5e4a libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5b5e libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b5e198000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b5e17e000 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b5df94000) /lib64/ld-linux-x86-64.so.2 (0x00007f5b5e4f5000) Links are just other files loaded into memory!
  4. A scripting language starts with a binary…. That loads instructions

    and interprets/optimizes/executes high-level instructions myscript.sh node index.js ruby app.rb 5
  5. Sometime scripts are compiled to an intermediate language (as a

    library) and interpreted/optimized/executed... dotnet myapp.dll java myapp.jar wasm myapp.wasm Still requires linked libraries! 6
  6. Sometimes you can have just a binary… Statically compiled… With

    dependencies… CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags '-w -extldflags "-static"' -o mybin *.go 7
  7. In The Beginning • Hardware Ruled The World • Infrastructure

    Was Static • Processes Were bad • Vendors Were Dictators • Things Took Months/Years 8
  8. 9 F5 IIS APACHE DB SAN/RAID Apache with: mod_perl mod_php

    phusion passenger IIS Java (+Apache) with: JBoss Websphere Tomcat VMWare Public IP Private DNS
  9. 13 But there were still problems • Did not solve

    shared libraries • Did not solve configuration • Routing had limitations • Did not solve deploy
  10. 14 Immutable Infrastructure My first public talk! • Use Packer

    • Bake Images • Launch with automation • CFN, ASGs, ELBs, R53, S3, EBS • Jenkins, a UI for bash
  11. 15 And all was good in the world. But darkness

    lurked... • Things were slow • Hard to debug • Bespoke automation
  12. 16 And Then There Was Docker Which is just a

    binary. With its linked libraries. Bundled in a tarball. With better automation. And easier process management.
  13. 17 Containers Everywhere Architecture rethought. 3 tier, fat process ->

    Microservices Back to square one… CoreOS Mesosphere Kubernetes Docker Swarm Jaeger Istio... Lots of processes Lots of ad-hoc automation
  14. 19 Requirements • We need servers (maybe not?!?!) ◦ We

    need to run processes • We need to configure these things • We need to install binaries on these ‘things’ • We need to configure these binaries • We need to run these binaries as processes • We need to secure and manage these things and processes • We need to route traffic to these processes • We need to update these things • We need to update these processes How how how?!?!?!!?
  15. 20 Observability Continuous Integration Continuous Delivery/Deployment Runtime (Kubernetes/Docker/App Servers) Server

    Infrastructure (AWS) Operations Configuration Management Environment Management Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs Security
  16. There is no right answer, only various degrees of wrong.

    We experiment, learn, decide, act, rinse, repeat and improve!
  17. 22 SignalFx, Logz.io, New Relic Jenkins Spinnaker/Octopus Kubernetes + Istio

    AWS Spinnaker, kubectl, Kubernetes, ad-hoc Kubernetes/Octopus Terraform Foundational Infrastructure Ease of Development, Testing and Delivery Meeting Production SLOs IAM, RBAC, Networking, VPN, Secrets
  18. 23 Welcome to Namely Infra AWS (Virginia) Production Int Stage VendorX Ops Portal IT An environment is: • An AWS account and permissions • A VPC • Route tables • Everything required to run Namely • The ability to deploy components
  19. Easy CIDR: A little lesson in IPs and

    An IP4 address is 32 bits That’s 32 1’s and 0’s 11110000 11110000 11110000 11110000 The /8 and /16 denotes how many bits ‘to keep’ (Big E) This denotes how IP’s are allocated A route table directs an IP range to a target
  20. 25 Environment Basics a Peering Public ELB(s) Internet Gateway Server1

    Server2 Jumpboxes Server3 VPC A bunch of RDS Alotta ElasticCache Some Aurora CloudFront S3 Kubernetes 15 workers 3 masters 5 etcd
  21. 26 Not so simple, though Zone 1a, private subnet, @ Zone 1c, private subnet, Zone 1d, private subnet, @ Node Node Node Node Node Node Node Node Node Zone 1a, public subnet, Zone 1c, public subnet, Zone 1d, public subnet, Public ELBs With Public IPs Private ELBs With Subnet IPs Try using the dig command to find out how dns names are mapped to IPs ENI ENI ENI SG SG SG SG SG SG SG SG SG
  22. 28 Kubernetes • Allows us to run containers ◦ A

    container is just a binary, with its dependencies • Abstracts node-management issues for processes ◦ IP addresses, ports, security, quotas ◦ Also in this space: ECS, Docker-Swarm, GAE • Built-in config, secret, service discovery, scaling management • Automates some cloud infra, like load balancers • Foundational-focused API (doesn’t solve some things) ◦ Allows for extensibility
  23. 29 Kubernetes Cluster Etcd0 Etcd1 Etcd2 Etcd3 Etcd4 Master0 Master1

    Master2 Worker0 Worker2 Worker3 Worker1 Worker... Worker15 State is stored here Does most of the k8s work Where stuff runs
  24. 31 Kubernetes Pods • The smallest unit that runs on

    a cluster • Has a name and labels • Has one, unique private IP address • Has one or more containers • All containers in a pod share networking, storage ◦ Can “see” eachother on localhost Pod Name: slug-bcddbcd8-1sa3a Labels: • app: slug Containers: • Slug • image: namely/slug • ports: ◦ 50051 • env • Istio • A ‘Sidecar’ is simply another container in a pod, usually auto-injected
  25. 32 32 32 Kubernetes Networking Kubernetes creates a private address

    space for pods, and handles routing across nodes Worker0 Worker1 Worker2 Pod0 Pod1 Kubelet Pod0 Pod1 Kubelet Pod0 Pod1 Kubelet
  26. 33 33 Kubernetes ReplicaSets • ReplicaSets create Pods • They

    specify how many instances of a pod should be running • Represents desired state ◦ Kubernetes will try to get current state to desired state • You usually don’t deal with these though • But they are why Pods are re-created after you delete them! ReplicaSet Name: slug-bcddbcd8-1sa3a Labels: • app: slug Spec: • Replicas: 2 • Selector ◦ Labels • Template ◦ Same as Pod! Pod Pod
  27. 34 Kubernetes Deployments • Deployments Manage ReplicaSets • You usually

    deal with these ◦ Through Spinnaker! • Deployments wind down old ReplicaSets and scale up new ones. • They support various strategies for how things are updated. Deployment Name: slug Labels: • app: slug Spec: • Replicas: 4 • Strategy ◦ Rolling-update • Selector ◦ Labels • Template ◦ Same as Pod! RS-old RS-new
  28. Other ways to run Pods • CronJobs • DaemonSets •

    Jobs • kubectl run • kubectl exec lets you ‘get into’ a pod
  29. Kubernetes Services • The ‘default’ way pods discover pods (L4)

    • Uses an internal DNS service ◦ Currently CoreDNS • Also uses a private network (10.3/16) • Uses labels to match service names with pods • Has three types. ◦ ClusterIP (default) ◦ LoadBalancer ▪ Allows external traffic to flow to internal pods ▪ On AWS creates an ELB ◦ Don’t worry about type three Service Name: slug Labels: • app: slug Spec: • Type • Ports • Selector ◦ Labels
  30. Time for the OSI Stack!!!!!! Layer 7 Layer 5/6 Layer

    4 Layer 3 Layer 2 Layer 1 Application Blah Transport Network Data Link Physical HTTP/HTTP2 TCP/UDP IPv4/IPv6 Ethernet* Raw wires(less) This is important This is important
  31. Kubernetes Ingress • Services only speak L4-TCP/UDP ◦ ‘Limited’ routing

    (single service) • Ingress is L7. It knows about http and http2 • Not natively implemented by Kubernetes ◦ Only schema is defined ◦ Third parties implement ◦ We tried Contour, Nginx, and Istio • Allows us to compose, shape and route traffic declaratively ◦ Used for gRPC traffic ◦ Used for our APIs ◦ Used to better handle AATE egress Ingress Name: slug Labels: app: slug Spec: rules: - host: '*.i.namely.com' http: paths: - backend: serviceName: slug servicePort: 80 path: /api/slug
  32. Sadly not perfect • Load Balancing for gRPC • How

    do you determine ‘healthy’ pod • No standard metrics, tracing • No way to test failures • Can’t optimize traffic (inter-zone) • Standard retries, fail-fast
  33. Observability • Bypasses K8S Service ◦ No DNS • Speaks

    L4 and L7 • Emits stuff for us • Big plans for Namely ◦ A/B testing ◦ Failure testing ◦ Better traffic management Some Pod App Container Istio Other Pod App Container Istio Straight to Pod’s IP Hey K8S, tell me every service and every pod IP
  34. We Must Automate If we are engineering processes and solutions

    that are not automatable, we continue having to staff humans to maintain the system. If we have to staff humans to do the work, we are feeding the machines with the blood, sweat, and tears of human beings. Think The Matrix with less special effects and more pissed off System Administrators engineers.
  35. Automation is about finding the right level of abstraction Want

    to do things easily Want to standardize… Yet allow for customization Build knowledge Find good tools and happy path
  36. Kubernetes CRDs Allows you to extend the native Kubernetes API

    We will use this for declarative management of Namely-things (like estuary -> spinnaker) Service creation, secrets, Pager Duty, Dashboards, Databases, Redis Apply yaml just like Deployments, Services, Ingress (Service Catalog) NamelyService Name: employee Spec: Team: hcm Requirements: - Redis - Postgres Observability: - Bugsnag - PagerDuty - SignalFx