What can servicing Kubernetes tell us about the future of open source in the cloud?

Enabling, integrating, releasing and contributing to open source have been the four pillars of Microsoft's approach to open source in the cloud. How can an organization like Microsoft learn from developers, customers and partners that use Kubernetes in our platform or with our products? And what does an operational consideration such as the servicing of a Kubernetes cluster tell us about the changing preferences and upcoming challenges with open source in the Enterprise? This talk was given at the "Road to FOSDEM" Meetup in Rotterdam on January 2020.

Transcript

1. What can servicing Kubernetes tell us about the future of open source in the cloud? Jose Miguel Parrella Office of the Azure CTO, Microsoft @bureado
2. Open source at Microsoft: a cultural change driven by demographics and leadership affinity Phase I: 2000- 2005 • "Shared Source" • "Accidental" product truths (Interix) Phase II: 2005- 2010 • CodePlex • "Insular" product truths (PHP on Windows, but also Linux on Hyper-V) Phase III: 2010- 2015 • "Trying too hard" • Microsoft Open Technologies • Node.js, TypeScript Phase IV: 2015- 2020 • Collaborative • Linux: Canonical, Red Hat • Hadoop: Hortonworks, Cloudera Phase V: Tomorrow • Innovative • Docker & Kubernetes • Rust & Golang Windows Azure Microsoft Azure
3. https://dev.to/bureado/ open-source- engagement-snow- tracks-2lkh
4. Open Source Tools & artifacts Roadmap Design Dialogue Community
7. How we are learning from Kubernetes • We run it as a service (AKS) • You can run it too (AKS Engine, Arc) • We brought Windows to it • ...and SQL Server • We believe in a great developer experience for it • We run on it • ...Xbox's Xcloud • Kubernetes is a driver of Linux usage, cloud-native development practices and deep community interaction for teams at Microsoft
8. Kubernetes servicing blows my mind • I'm a 15+ year Linux user that grew up in Latin America: • I lived the CFEngine to Puppet transition • I embraced (deb) packages as atomic units of delivery • Things like Corosync, Ultramonkey, etc., were basic clustering building blocks • Databases and NFS were best delegated elsewhere • We had maintenance windows • In Kubernetes, a node is the servicing unit – sometimes even the whole cluster. (this is what modern compute and network buys!) • In AKS, fresh nodes are added to the pool, then a node is cordoned and drained. If your state and ingress are outside your cluster, you might be able to just deploy a new cluster and point stuff at it.
10. Kubernetes servicing blows my mind • This means that Enterprise customers are now: • Adopting open source software at a much faster cadence than their regular enterprise open source vendors • Willing to take supply chain risks as final nodes are assembled from bits that are often coming from many different sources • Incurring the cost of replacing nodes even for trivial changes to achieve the benefits of immutability • Willing to introduce tools incompatibility as cluster API leaps forward (and in a DevOps world, "tools" can be anything from CD logic to operational sensors) • Seeing Kubernetes multiple personalities: scheduler, state driver, app plat... • Note: Kubernetes versions n-4, n-5 and n-6 are highly prevalent • Lawrence Hecht on Twitter: most popular versions are 10+ months old, "...users of hosted Kubernetes services are usually presented with a default, stable version at cluster launch...version 1.13 was still the default for GKE and AKS, whereas EKS provided version 1.14 by default."
11. We all think Kubernetes is "going away" but it's hard to tell what it will look like in 3 years • Will it be defined by the application model? • Will it be defined by the application packaging and delivery? • Will it be defined by the service mesh? • Will it be driven by paradigms such as event-driven, dataflow, etc.? • Will it be driven by where the applications and data live? • Will it be driven by innovation in the underlying virtualization and container technologies? • Will it just be "servicefied" by a handful of service providers and that's it?
12. Predictions in the wild • Part-time developers such as business analysts, data analysts and data scientists to double in five years (as a reference, there's 40M people on GitHub, but only 25M devs according to Evans Data...) • By 2024, low-code application development will be responsible for more than 65% of application development activity (Gartner) • Use of open source software from public repositories used to build enterprise apps to double, and... • By 2023, 60% of the G2000 enterprises will have created their own software ecosystem, while 90% will use publicly accessible community repositories to access reusable code components (IDC) • By 2023, 60% of the G2000 will have a digital developer ecosystem with thousands of developers; half of those enterprises will drive 20%+ of digital revenue through their digital ecosystem/platform (also read)
13. Stuff I'm watching... • We can still do better at persisting state • From operators et al to File systems unfit as distributed storage backends • Confidential computing and enclaves • sgx-lkl • Software supply chain security and Kubernetes • How data flows in and out of pods, licensing, model pedigree... • Kubernetes operators that are bringing learnings to Linux • Yelp's ClusterMan, Canonical's Dqlite, HashiCorp's Nomad, Kong's Kuma, Puppet's Bolt, GitHub's GLB, Clockwork-Scheduler (DynamoDB)
14. PSI ◦ Quantifies lost wall clock time due to resource contention ◦ Exported via /proc ◦ Works with cgroup2 ◦ In kernel 4.20 and newer ◦ Full introduction of PSI in the kernel mailing lists Jose Miguel Parrella | @bureado
15. Stuff I'll be watching during FOSDEM • Containers and security • Containers devroom • Storage devroom • Lazy distribution of container images • Linux memory management at scale • Do Linux Distributions Still Matter with Containers? • Demo: SGX-LKL • Ephemeral Environments For Developers In Kubernetes • Immutable deployments: the new classic way for service deployment • Kubernetes of Things • Linux Kernel Library • PostgreSQL on K8S at Zalando: Two years in production
16. ...all of this isn't limited to Kubernetes! • It's happening with Linux... • ...for storage: io_uring, bcachefs • ...for servicing: Yocto in automotive/embedded • It's happening in the container ecosystem at large with software supply chain with things like TUF, in-toto and more... • It's happening in the Node.js, PyPI and Golang ecosystems for things like software supply chain... • It's happening with the explosion of innovation in open source databases, including multi-model DBs and licensing experiments...
17. Debian at Microsoft APT is (possibly) the most prevalent package manager in the Microsoftecosystem Microsoft distributes software for Debian Microsoft runs Debian for top-of-rack networking in all datacenters Microsoft runs Debian (on Azure) for all the Skype relays Microsoft builds a minimal set of Debian packages for reproducibility Jose Miguel Parrella | @bureado
18. jose.miguel@microsoft.com @bureado speakerdeck.com/bureado