The (long) road to Kubernetes

Transcript

1. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 1/67
   THE (LONG) ROAD TO
   THE (LONG) ROAD TO
   KUBERNETES
   KUBERNETES
   1
   

   View Slide

2. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 2/67
   @PYR
   @PYR
   CTO, co-founder at
   Exoscale
   Open source developer
   2 . 1
   

   View Slide

3. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 3/67
   OUTLINE
   OUTLINE
   A small intro for context
   The road to distributed systems
   Keeping promises in a containerized
   world
   3 . 1
   

   View Slide

4. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 4/67
   EXOSCALE
   EXOSCALE
   Infrastructure as a service
   Part of A1 Digital
   Zones in Frankfurt, Vienna, Zürich, Geneva, more in
   2019!
   4 . 1
   

   View Slide

5. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 5/67
   EXOSCALE
   EXOSCALE
   5 . 1
   

   View Slide

6. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 6/67
   EXOSCALE
   EXOSCALE
   provider "exoscale" {
   api_key = "${var.exoscale_api_key}"
   secret_key = "${var.exoscale_secret_key}"
   }
   resource "exoscale_instance" "web" {
   template = "Ubuntu 18.04"
   disk_size = "50g"
   profile = "medium"
   ssh_key = "production"
   }
   6 . 1
   

   View Slide

7. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 7/67
   I THOUGHT THIS WAS A CONTAINER
   I THOUGHT THIS WAS A CONTAINER
   CONFERENCE!
   CONFERENCE!
   7 . 1
   

   View Slide

8. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 8/67
   WHAT'S IN A CLOUD PROVIDER
   WHAT'S IN A CLOUD PROVIDER
   Datacenter operations
   So ware
   development
   8 . 1
   

   View Slide

9. 2/1/2019 The (long) road to Kubernetes
   http://localhost:8000/pres.html?print-pdf#/sec-title-slide 9/67
   SOFTWARE AT EXOSCALE
   SOFTWARE AT EXOSCALE
   Object storage
   controller
   Network controller
   Internal SDN
   Customer management
   Metering system
   Billing
   Web portal
   9 . 1
   

   View Slide

10. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 10/67
    LANGUAGES AT EXOSCALE
    LANGUAGES AT EXOSCALE
    C & Go
    Clojure
    Python
    ClojureScript &
    JS
    10 . 1
    

    View Slide

11. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 11/67
    THE ROAD TO DISTRIBUTED SYSTEMS
    THE ROAD TO DISTRIBUTED SYSTEMS
    11 . 1
    

    View Slide

12. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 12/67
    IT ALWAYS STARTS WITH A SIMPLE PRODUCT
    IT ALWAYS STARTS WITH A SIMPLE PRODUCT
    You want to change the world by disrupting the job board
    industry
    Standard three-tier, self-contained app
    Does not fall into the usual definition of distributed systems
    12 . 1
    

    View Slide

13. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 13/67
    IT ALWAYS STARTS WITH A SIMPLE PRODUCT
    IT ALWAYS STARTS WITH A SIMPLE PRODUCT
    13 . 1
    

    View Slide

14. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 14/67
    FIRST SIGNS OF SUCCESS
    FIRST SIGNS OF SUCCESS
    Your single server is not sufficient anymore
    Database gets its own machines, adding new web servers fixes the
    issue
    Logging becomes a bit harder
    You switch to a centralized logging solution
    14 . 1
    

    View Slide

15. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 15/67
    FIRST SIGNS OF SUCCESS
    FIRST SIGNS OF SUCCESS
    15 . 1
    

    View Slide

16. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 16/67
    FEATURES GET ADDED
    FEATURES GET ADDED
    Subscriptions emails
    Doing it synchronously is impossible
    Let's add a worker (and thus a queueing
    mechanism)
    You start switching from pets to cattle
    16 . 1
    

    View Slide

17. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 17/67
    FEATURES GET ADDED
    FEATURES GET ADDED
    17 . 1
    

    View Slide

18. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 18/67
    SEED MONEY RUNS OUT
    SEED MONEY RUNS OUT
    Let's try other monetization techniques
    Freemium model with analytics
    Where do I run these batch jobs?
    You partner with another company to exchange data
    They have this weird legacy system and the only client lib is in
    PHP :-(
    18 . 1
    

    View Slide

19. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 19/67
    SEED MONEY RUNS OUT
    SEED MONEY RUNS OUT
    19 . 1
    

    View Slide

20. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 20/67
    AS THE PRODUCT GROWS, SO DOES
    AS THE PRODUCT GROWS, SO DOES
    INFRASTRUCTURE
    INFRASTRUCTURE
    You're now at 3 jenkins workers
    You had to split metrics and monitoring on separate machines
    You introduce a command and control solution to perform your
    regular operations
    It's time to use puppet (You're really starting to feel like an ops
    person now)
    20 . 1
    

    View Slide

21. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 21/67
    LET'S TAKE A STEP BACK
    LET'S TAKE A STEP BACK
    You're ticking all the boxes: CI, Infrastructure as Code,
    DevOps
    21 . 1
    

    View Slide

22. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 22/67
    BUT RESOURCE UTILIZATION IS LOW
    BUT RESOURCE UTILIZATION IS LOW
    Most of it is articifial (agents on every nodes)
    But you still have peak induced regular
    contention
    22 . 1
    

    View Slide

23. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 23/67
    ADDING NEW SERVICES OR COMPONENTS IS HARD
    ADDING NEW SERVICES OR COMPONENTS IS HARD
    Should your most active git repository really be the puppet
    one?
    You constantly have to make allocation decisions
    23 . 1
    

    View Slide

24. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 24/67
    HANDLING FAILURE IS HARD
    HANDLING FAILURE IS HARD
    Your monitoring system tells you when something breaks
    You have to recreate machines manually, update configuration all
    over the place
    24 . 1
    

    View Slide

25. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 25/67
    FROM A SERVICE POINT OF VIEW IT ALL MAKES
    FROM A SERVICE POINT OF VIEW IT ALL MAKES
    SENSE
    SENSE
    25 . 1
    

    View Slide

26. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 26/67
    WHAT WOULD BE NICE
    WHAT WOULD BE NICE
    26 . 1
    

    View Slide

27. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 27/67
    WHAT WOULD BE NICE
    WHAT WOULD BE NICE
    27 . 1
    

    View Slide

28. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 28/67
    WHERE TO FROM HERE
    WHERE TO FROM HERE
    How do you get out of the business of shuffling configuration and
    apps around?
    28 . 1
    

    View Slide

29. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 29/67
    ALTERNATIVES CONSIDERED
    ALTERNATIVES CONSIDERED
    Mesos
    Docker
    LXD
    Kubernetes
    29 . 1
    

    View Slide

30. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 30/67
    ALTERNATIVES CONSIDERED
    ALTERNATIVES CONSIDERED
    Mesos
    Docker
    LXD
    Kubernetes
    30 . 1
    

    View Slide

31. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 31/67
    KUBERNETES PROMISES
    KUBERNETES PROMISES
    Utilization optimization
    Reducing the pain of adding new
    components
    Command and control
    Preserves a service boundaries
    31 . 1
    

    View Slide

32. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 32/67
    ADDITIONAL PROMISES
    ADDITIONAL PROMISES
    A good substrate for creating resources on-
    demand
    Lingua-franca for infrastructure concepts
    Initial learning-curve
    Most likely smaller than ad-hoc solutions
    Eating our own dog food
    32 . 1
    

    View Slide

33. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 33/67
    LET'S INSTALL KUBERNETES AND GO
    LET'S INSTALL KUBERNETES AND GO
    HOME?
    HOME?
    33 . 1
    

    View Slide

34. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 34/67
    REMAINING WORRIES
    REMAINING WORRIES
    Security
    Monitoring
    Deployment
    Process
    Networking
    34 . 1
    

    View Slide

35. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 35/67
    DEPLOYMENT PROCESS
    DEPLOYMENT PROCESS
    Reproducibility
    Traceability
    Security
    Checkpoints
    35 . 1
    

    View Slide

36. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 36/67
    REPRODUCIBILITY
    REPRODUCIBILITY
    Building once (and in chroots) ensures clean packages
    Reproducible builds make wide changes easier
    We need staging deploys and production deploys to be
    identical
    36 . 1
    

    View Slide

37. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 37/67
    TRACEABILITY
    TRACEABILITY
    When did we last build this?
    What did the output look like?
    What commit did it correspond
    to?
    37 . 1
    

    View Slide

38. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 38/67
    SECURITY
    SECURITY
    No code download on production
    hosts
    Signed packages
    38 . 1
    

    View Slide

39. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 39/67
    CHECKPOINTS
    CHECKPOINTS
    CD is great for test and staging
    We are wary of unattended production
    deploys
    There should be a clear (but simple) trigger
    39 . 1
    

    View Slide

40. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 40/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    40 . 1
    

    View Slide

41. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 41/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    41 . 1
    

    View Slide

42. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 42/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    42 . 1
    

    View Slide

43. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 43/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    43 . 1
    

    View Slide

44. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 44/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    44 . 1
    

    View Slide

45. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 45/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    45 . 1
    

    View Slide

46. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 46/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    46 . 1
    

    View Slide

47. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 47/67
    AFTER KUBERNETES
    AFTER KUBERNETES
    47 . 1
    

    View Slide

48. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 48/67
    AFTER KUBERNETES
    AFTER KUBERNETES
    Reproducibility
    ✔
    Traceability ✔
    Security ✔
    Checkpoints ✔
    48 . 1
    

    View Slide

49. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 49/67
    AFTER KUBERNETES: PACKAGING
    AFTER KUBERNETES: PACKAGING
    Building is faster, easier, and gives developers more autonomy
    Docker registries forced us to move from enforcement to
    convention
    49 . 1
    

    View Slide

50. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 50/67
    AFTER KUBERNETES: CONFIGURATION
    AFTER KUBERNETES: CONFIGURATION
    The split across environment services, variables, configmaps, and
    secrets makes separation easy
    Greatly reduces the need for config management
    Configuration can be colocated with the so ware
    Removes the code, build, and configuration-management
    impedance mismatch
    Kept things simple
    No helm
    50 . 1
    

    View Slide

51. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 51/67
    AFTER KUBERNETES: ON-DEMAND
    AFTER KUBERNETES: ON-DEMAND
    RESOURCES
    RESOURCES
    CRDs provide great integration
    High cardinality or complex queries can be tedious to work
    with
    51 . 1
    

    View Slide

52. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 52/67
    AFTER KUBERNETES: SECURITY
    AFTER KUBERNETES: SECURITY
    RBAC policies are powerful but tedious to write (and error
    prone)
    Certificate management leaves a lot to be improved
    52 . 1
    

    View Slide

53. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 53/67
    GOING FORWARD
    GOING FORWARD
    Going back from convention to enforcement for the
    registry
    Providing more Paas-like encoding of our common cases
    53 . 1
    

    View Slide

54. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 54/67
    NETWORKING
    NETWORKING
    Security
    Scalability
    54 . 1
    

    View Slide

55. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 55/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    A public IP per VM
    Security groups for firewall
    management
    55 . 1
    

    View Slide

56. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 56/67
    BEFORE KUBERNETES
    BEFORE KUBERNETES
    A boring, solid network
    56 . 1
    

    View Slide

57. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 57/67
    KUBERNETES NETWORKING 101
    KUBERNETES NETWORKING 101
    57 . 1
    

    View Slide

58. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 58/67
    KUBERNETES NETWORKING 101
    KUBERNETES NETWORKING 101
    58 . 1
    

    View Slide

59. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 59/67
    KUBERNETES NETWORKING 101
    KUBERNETES NETWORKING 101
    59 . 1
    

    View Slide

60. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 60/67
    KUBERNETES NETWORKING 101
    KUBERNETES NETWORKING 101
    A boring, solid network
    60 . 1
    

    View Slide

61. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 61/67
    WHAT ABOUT EXTERNAL SERVICES
    WHAT ABOUT EXTERNAL SERVICES
    By default NodePort services run on all worker nodes
    Traffic is source-nat'd to the destination
    Losing source IP information is unviable for most use-
    cases
    Performance impact
    61 . 1
    

    View Slide

62. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 62/67
    NETWORKING AT EXOSCALE
    NETWORKING AT EXOSCALE
    A layer3 all-the-way design
    A BGP first design
    VM Public IPs advertised by BGP from hypervisors
    Private network VXLAN membership advertised through BGP-
    eVPN
    Best performance is on the public interfaces
    62 . 1
    

    View Slide

63. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 63/67
    NETWORKING: AFTER KUBERNETES
    NETWORKING: AFTER KUBERNETES
    Security ✔
    Scalability
    ✔
    63 . 1
    

    View Slide

64. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 64/67
    IN-CLUSTER NETWORKING
    IN-CLUSTER NETWORKING
    Kept with a BGP-based CNI
    Avoids additional
    encapsulation
    64 . 1
    

    View Slide

65. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 65/67
    EXTERNAL SERVICES
    EXTERNAL SERVICES
    Needs additional development
    We went for IPIP load-balancing
    Needs node-local
    decapsulation
    Makes for a hacky setup
    eBPF/XDP to the rescue!
    65 . 1
    

    View Slide

66. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 66/67
    KUBERNETES AT EXOSCALE TODAY
    KUBERNETES AT EXOSCALE TODAY
    Most production critical services
    Some things we are in no hurry to containerize
    :-)
    Basis for as-a-service offerings
    Private network management
    For customers
    exo lab kube
    Cluster API
    OpenShi
    On-demand cluster in the Exoscale API
    66 . 1
    

    View Slide

67. 2/1/2019 The (long) road to Kubernetes
    http://localhost:8000/pres.html?print-pdf#/sec-title-slide 67/67
    QUESTIONS?
    QUESTIONS?
    We're productizing all of this as we speak!
    67 . 1
    

    View Slide