The Journey to #GIFEE

Walking through the 3 changes required to run a Google-like infrastructure stack on CoreOS + Kubernetes.

Transcript

1. Rob Szumski
   @robszumski | rob.szumski@coreos.com
   The Journey to #GIFEE
   

   View Slide

2. Secure the Internet
   MISSION
   

   View Slide

3. View Slide

4. 3
   Application packaging
   Linux at scale
   Clustering
   

   View Slide

5. #GIFEE
   Borg/Omega
   ChromeOS
   Chubby
   

   View Slide

6. #GIFEE
   Borg/Omega
   ChromeOS
   Chubby
   

   View Slide

7. #GIFEE
   Borg/Omega
   ChromeOS
   Chubby
   

   View Slide

8. Linux at Scale
   1
   

   View Slide

9. Patches to the OS and kernel are hard
   Rolling update tools
   Diverse hardware
   LARGE SCALE
   Safer to leave it alone
   No one owns security
   SMALL SCALE
   

   View Slide

10. Auto-updating browsers fixed security
    We got HTML5 at the same time
    

    View Slide

11. Atomic operating system updates
    

    View Slide

12. Atomic operating system updates
    

    View Slide

13. PXE/diskless
    Quick reboots
    Easy to boot, install, and manage
    Secure by default
    Cross-cloud
    

    View Slide

14. View Slide

15. Application Packaging
    2
    

    View Slide

16. Abstract away app from the OS
    OS App
    

    View Slide

17. View Slide

18. View Slide

19. Protect apps from each other
    Isolated network namespace
    Isolated file system namespace
    Mixed versions of dependencies
    eg. python 3.4 & python 2.7
    

    View Slide

20. Base software managed by CoreOS
    systemd kernel OpenSSH
    

    View Slide

21. Easily move apps between machines
    Easy scale out
    Recover from failure
    Painless OS software update
    

    View Slide

22. Perfect touch-point for security
    Sign artifact from CI
    Scan containers at rest
    Audit trail
    

    View Slide

23. View Slide

24. A security-minded, standards-based
    container engine
    

    View Slide

25. Specification for “application containers”
    

    View Slide

26. Universal Container Format
    Packaged Downloaded Executed
    

    View Slide

27. apt-get for containers
    Local mirrors
    Distributed namespace (DNS)
    Serve over HTTPS, no complex software
    

    View Slide

28. Jetpack
    FreeBSD/Go
    Kurma
    Linux/Go
    rkt
    Linux/Go
    Independent GitHub organization
    Contributions from Cloud Foundry,
    Mesosphere, Google, Red Hat and many others
    

    View Slide

29. View Slide

30. Composable
    Designed for init systems
    Standard Unix process
    Separate build tool
    

    View Slide

31. Composable
    No central daemon
    Not a “platform”
    

    View Slide

32. systemd
    app
    systemd
    app
    docker run redis
    docker engine daemon
    

    View Slide

33. $ sudo rkt run coreos.com/etcd:v2.0.0
    $ sudo rkt run coreos.com/etcd:v2.0.0 \
    --cpu=750m --memory=128M
    $ sudo rkt run --net=host coreos.com/etcd:v2.0.0
    rkt run
    

    View Slide

34. View Slide

35. Pods Built-in
    Deployed together
    Share local network
    Share volumes
    

    View Slide

36. Pods Built-in
    rktnetes
    

    View Slide

37. Tunable Isolation
    Match your workload
    3 isolation levels
    Make your own stage1
    

    View Slide

38. stage0
    stage1
    stage2
    

    View Slide

39. stage0
    stage1
    stage2
    The rkt binary
    ● Fetch ACI, verify
    ● Set up pod filesystem
    ● Unpack stage1 and
    stage2 ACIs
    

    View Slide

40. stage0
    stage1
    stage2 Set up execution env
    ● Create cgroups,
    namespaces, & mounts
    ● Read pod manifest
    ● Start systemd-nspawn
    

    View Slide

41. stage0
    stage1
    stage2 Your application!
    

    View Slide

42. Benefit from standard packaging, signing and distribution at all isolation levels.
    Privileged
    eg. Kubelet
    Container/cgroup
    eg. Webapp
    Virtual Machine
    eg. Untrusted code
    

    View Slide

43. $ sudo rkt run \
    example.com/worker -- --loglevel verbose --- \
    example.com/syncer -- --interval 30s
    rkt run a pod
    

    View Slide

44. Unique rkt features
    Sensible, best practice security
    Ease of use for Ops
    

    View Slide

45. $ sudo rkt gc
    Moving pod "81627cc6" to garbage
    Moving pod "cd642877" to garbage
    Moving pod "d65abad6" to garbage
    Pod "81627cc6" not removed: still within grace period
    (30m0s)
    Pod "cd642877" not removed: still within grace period
    (30m0s)
    Pod "d65abad6" not removed: still within grace period
    (30m0s)
    Garbage Collection
    Run as a cron job, customizable grace period
    

    View Slide

46. $ sudo rkt trust --prefix=storage.coreos.com
    $ sudo rkt trust --prefix=coreos.com/etcd
    $ sudo rkt trust --root ~/aci-pubkeys.gpg
    Tools for trust
    Easily control what runs on your server
    

    View Slide

47. $ find /etc/rkt/trustedkeys/
    /etc/rkt/trustedkeys/
    /etc/rkt/trustedkeys/prefix.d
    /etc/rkt/trustedkeys/prefix.d/coreos.com
    /etc/rkt/trustedkeys/prefix.d/coreos.com/etcd
    /etc/rkt/trustedkeys/prefix.d/coreos.
    com/etcd/8b86de38890ddb7291867b025210bd8888182190
    /etc/rkt/trustedkeys/root.d
    /etc/rkt/trustedkeys/root.
    d/d8685c1eff3b2276e5da37fd65eea12767432ac4
    Tools for trust
    Easily control what runs on your server
    

    View Slide

48. $ rkt fetch quay.io/coreos/alpine-sh
    ...
    $ sudo rkt run quay.io/coreos/alpine-sh
    Fetch ACI as unprivileged user
    Don’t have to download as root
    

    View Slide

49. $ sudo rkt run --insecure-options=image --interactive \
    docker://busybox -- /bin/sh
    Run Docker containers with rkt
    Use a more secure runtime without changing images
    

    View Slide

50. Clustering
    3
    

    View Slide

51. Scale out workloads
    Everyone’s goal is #GIFEE
    Enables automation
    Cloud = Distributed Systems
    

    View Slide

52. When do you need
    cluster coordination?
    Leader election Cluster-wide Semaphores
    Service discovery
    Dynamic configuration
    

    View Slide

53. Hard Computer Science Problem
    ?
    

    View Slide

54. Hard Computer Science Problem
    Chubby
    

    View Slide

55. A distributed, reliable key-value
    store for the most critical data of a
    distributed system.
    

    View Slide

56. View Slide

57. No existing “cloud native” solutions
    High availability from beginning
    Dynamic reconfiguration
    Why build etcd?
    

    View Slide

58. Simple key/value
    “Distributed etc”
    Feels like a file system
    eg. directories
    

    View Slide

59. $ etcdctl set /foo bar
    bar
    $ etcdctl ls /config
    /config/verbosity
    /config/ratelimit
    Set a value
    $ etcdctl get /foo
    bar
    

    View Slide

60. Simple interface
    Easily write clients
    Use curl if you want
    Already maintain TLS infra.
    

    View Slide

61. Watch a value
    Service discovery
    Reconfiguration
    Locking
    Cluster scheduler
    

    View Slide

62. Cluster-wide reboot lock - “locksmith”
    Distributed init system - “fleet”
    Leader election - “fleet”
    

    View Slide

63. $ locksmithctl status
    Available: 1
    Max: 1
    $ sudo locksmithctl reboot
    locksmith
    $ locksmithctl status
    Available: 0
    Max: 1
    MACHINE ID
    7f9ccde3cff9441f8b506785
    $ sudo locksmithctl reboot
    Error locking: semaphore is
    at 0
    

    View Slide

64. Industry Adoption
    500+ projects on Github
    

    View Slide

65. 3
    Application packaging
    Linux at scale
    Clustering
    

    View Slide

66. Minimal, secure Linux OS
    Containers for app packaging
    Self-updating cluster
    Distributed systems tools
    

    View Slide

67. View Slide

68. Sounds good, but...
    Is anyone successful with CoreOS in prod?
    

    View Slide

69. Publically traded options exchange
    

    View Slide

70. Containers on CoreOS are powering ISE's high-
    throughput, low-latency financial exchange
    Running in production
    Bare metal & AWS
    Billions of transactions a day
    150 million req/sec
    

    View Slide

71. TIME PATCHING OS
    NEW MACHINE DEPLOYMENT
    

    View Slide

72. Invisible
    Infrastructure
    

    View Slide

73. We really look at that [CoreOS] number growing
    significantly over this next year. We did some of these
    benchmarks to see if our production trading systems could
    leverage this type of infrastructure, and it was highly
    successful for us, and we look forward to using it more in
    our other environments.
    On the Linux side, everything in AWS is CoreOS. On the
    physical side, 20% is CoreOS, and growing.
    “
    ”
    Robert Cornish
    CTO
    Paul Morgan
    Systems Architect
    

    View Slide

74. View Slide

75. Kubernetes is our recommended
    orchestration platform
    

    View Slide

76. Guides & Tools
    coreos.com/kubernetes
    kube-aws
    Cloud-configs
    

    View Slide

77. Upstream
    rktnetes
    Auth/OIDC
    Node self-signed TLS
    

    View Slide

78. Scaling
    15x scheduler performance
    30k pods on 1k nodes
    SIG-scale
    

    View Slide

79. View Slide

80. Off-the-shelf
    #GIFEE
    

    View Slide

81. Enhances
    Kubernetes
    Included tools
    24/7 Support
    Enhanced security
    

    View Slide

82. Quay Enterprise
    

    View Slide

83. Tectonic Console
    

    View Slide

84. CoreUpdate
    

    View Slide

85. Distributed
    Trusted Computing
    Only possible with #GIFEE
    

    View Slide

86. Trusted Computing
    It’s in your pocket right now
    

    View Slide

87. Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    

    View Slide

88. Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    

    View Slide

89. Customer key embedded in the firmware
    Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    Kubernetes
    

    View Slide

90. Verify integrity of the OS release
    Customer key embedded in the firmware
    Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    

    View Slide

91. Verify integrity of the OS release
    Customer key embedded in the firmware
    Verify configuration state
    Verify images with trusted keys
    Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    

    View Slide

92. Verify integrity of the OS release
    Customer key embedded in the firmware
    Verify configuration state
    Verify images with trusted keys
    Only attested machines are allowed to join
    Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    

    View Slide

93. Verify integrity of the OS release
    Customer key embedded in the firmware
    Verify configuration state
    Verify images with trusted keys
    Only attested machines are allowed to join
    Kubernetes
    rkt
    CoreOS Linux
    Firmware & TPM
    Cluster
    Containers
    Hardware
    OS
    Tamper-proof
    audit log (TPM)
    

    View Slide

94. Identify Attacks
    Visibility into new classes of attacks
    Firmware OS Images Rootkits
    

    View Slide

95. Inverting DRM
    Your company is in control
    

    View Slide

96. You hold the keys
    Only software your company allows will run
    You are in control of the hardware
    Key
    

    View Slide

97. New Level of Security
    Run in third party or hostile data centers with zero trust
    Prevent invisible attacks
    Verifiable audit log for when things go wrong
    Putting you in control
    Your company is in cryptographic control your environment
    

    View Slide

98. The Journey to #GIFEE
    

    View Slide

99. coreos.com/fest - @coreosfest
    May 9 & 10, 2016 - Berlin, Germany
    

    View Slide

100. Thank You
     Rob Szumski
     Product Design Lead, CoreOS
     @robszumski
     

     View Slide