$30 off During Our Annual Pro Sale. View Details »

SUSE Labs Conference 2020 - Thinking with a "Rolling First" mindset

SUSE Labs Conference 2020 - Thinking with a "Rolling First" mindset

While SUSE currently offers products only with some kind of regular release cadence, it is increasingly common for software projects of various sizes to have development & support models that are more easily described as 'rolling'. This phenomenon is impacting the expectations we now see coming from our customers and partners, and thusly does impact even our most conservative and long-legged products. This talk will discuss some of the benefits and weaknesses of this reality, and posit some opportunities if we adopted a more 'rolling centric' mindset in how we develop, build, and possibly eventually release our products and software components.

Richard Brown

October 06, 2020
Tweet

More Decks by Richard Brown

Other Decks in Programming

Transcript

  1. Thinking with a "Rolling First"
    mindset
    Considering priorities for the engineering and
    customer needs of today
    October 06, 2020, The Internet

    View Slide

  2. Copyright © SUSE 2020
    2
    Who am I?
    openSUSE contributor since it began
    SUSE employee since 2013
    Passionate advocate of rolling releases
    Linux Distribution Engineer in Future Technology Team
    focusing on two rolling distributions
    openSUSE MicroOS – Single Purpose Self Administering OS
    openSUSE Kubic – MicroOS with Kubernetes & Containers

    View Slide

  3. 3 Copyright © SUSE 2020
    Agenda
    01 Changing Upstream Expectations
    02 Changing Customer Expectations
    03 We Already Roll
    04 Engineering Assumptions & Pitfalls
    05 Engineering “Rolling First”
    06 Releasing “Rolling First”

    View Slide

  4. 4 Copyright © SUSE 2020
    Upstreams

    View Slide

  5. 5 Copyright © SUSE 2020
    Preaching to the
    Choir

    View Slide

  6. Copyright © SUSE 2020
    6
    Upstreams are fast
    Upstream software projects move very fast
    Glibc – New version every 6 months
    Linux Kernel – New version every 3 months
    Kubernetes – New version every 3 months
    SaltStack – New version every 3-6 months
    Uyuni – New version every 1-2 months
    Ceph – New version every 1-2 months
    Podman/skopeo/buildah – New version all the time
    Cloud Foundry – New versions all the time

    View Slide

  7. Copyright © SUSE 2020
    7
    Short Upstream Support Periods
    Upstreams rarely care for long support lifecycles
    Kernel Stable Releases – 4 months
    Kernel LTS Releases – 6-7 years
    Kubernetes – 1 year (as of v1.19, was 9 months for earlier versions)
    SaltStack – 1.5 years (feature freeze after 6 months)
    Ceph – 2 years

    View Slide

  8. Copyright © SUSE 2020
    8
    API/ABI Compatibility
    Upstreams rarely have long lasting API/ABI Compatibility
    Linux Standard Base – is anyone LSB 5.0 certified?
    Kubernetes – at least 12 months (admin-facing CLI interfaces 6 months)

    View Slide

  9. Copyright © SUSE 2020
    9
    More Upstreams
    We’re not just a Linux company any more
    SUSE Linux Enterprise (and family)
    SUSE Storage
    SUSE Manager
    SUSE CaaSP
    SUSE CAP
    and more to come...

    View Slide

  10. Copyright © SUSE 2020
    10

    View Slide

  11. Copyright © SUSE 2020
    11
    Bigger Upstreams - Kubernetes

    View Slide

  12. Copyright © SUSE 2020
    12
    Bigger Upstreams - Kubernetes

    View Slide

  13. Copyright © SUSE 2020
    13
    Bigger Upstreams - Kernel
    Source: phoronix.com “The Linux Kernel Enters 2020 At 27.8 Million Lines In Git But
    With Less Developers For 2019”

    View Slide

  14. Copyright © SUSE 2020
    14
    A More Efficient SUSE
    SUSE historically has never grown as fast as our upstream codebases
    SUSE is engaging with more new upstreams at a faster rate than ever
    before
    Customers are increasingly aware and interested in upstream
    developments
    We need to rise to these emerging challenges

    View Slide

  15. 15 Copyright © SUSE 2020
    Customers

    View Slide

  16. Copyright © SUSE 2020
    16
    Market Share
    2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
    Year
    0.00%
    0.05%
    0.10%
    0.15%
    0.20%
    0.25%
    0.30%
    0.35%
    0.40%
    % of Stack Overflow questions that month
    Tag
    ubuntu
    centos
    redhat

    View Slide

  17. Copyright © SUSE 2020
    17
    Market Share

    View Slide

  18. Copyright © SUSE 2020
    18
    Market Share

    View Slide

  19. Copyright © SUSE 2020
    19
    Reminder
    Red Hat Enterprise Linux – 10 year support (3 years less than SUSE
    Linux Enterprise)
    Ubuntu – 5 year support (8 years less than SUSE Linux Enterprise)
    Our more popular competition support things for shorter than we do

    View Slide

  20. Copyright © SUSE 2020
    20
    Complex Products
    Not only are our products more complex, but the way customers use it is more
    complex
    Increasingly rare that customers buy a single SUSE product as the solution to their
    problem
    Increasingly common that SUSE products are purchased as part of a wider
    solution, including external dependencies wholly out of our control, influence, and
    often knowledge
    Interoperability is key

    View Slide

  21. Copyright © SUSE 2020
    21
    Interoperability in Complex Solutions
    If SUSE products are just a cog in a much larger customer machinery,
    how likely is it that customers will decide all their other dependent
    technologies to match their choice of SUSE?
    Adaptability to customers other choices is required for Interoperability
    We can’t adapt if the codebase is frozen

    View Slide

  22. Copyright © SUSE 2020
    22
    Adoption of New Technologies
    There are 5 categories of innovation adopters
    Innovators (2.5%) - most willing to take risks, most financially fluid
    Early Adopters (13.5%)
    Early Majority (34%)
    Late Majority (34%)
    Laggards (16%) - least willing to take risks, least financially fluid
    (Rogers, Diffusion of Innovations 5th Ed.)

    View Slide

  23. Copyright © SUSE 2020
    23
    Adoption of New Technologies

    View Slide

  24. Copyright © SUSE 2020
    24
    Requests with “Very High Customer Interest”
    jsc#PM-2107 “Haproxy 1.8 on SLES 12SP5 HA”
    jsc#PM-1904 “Switch to Firefox 78 ESR release stream”
    jsc#PM-1835 “Upgrade mod_nss to enable TLSv1.3”
    jsc#PM-1804 “Integrate WireGuard into Kernel”
    jsc#PM-155[5, 6, 7] Tensorflow1, Tensorflow2, Pytorch (packages and containers)
    jsc#PM-1443 “Update protobuf to 3.6 or later”
    jsc#PM-1386 “git for SLES 12 with SHA 256 support”
    jsc#PM-1332 “Clustertools2 in SLESforSAP are outdated. Clients use the version from
    GitHub”
    jsc#PM-1237 “Update SLURM to version 19.05.1-2 or later”
    jsc#PM-1090 “Select and provide an updated kernel for 15 SP2”

    View Slide

  25. Copyright © SUSE 2020
    25
    Update SLURM to version 19.05.1-2 or later
    “HPC customers often require the latest version of numerical libraries.”

    View Slide

  26. 26 Copyright © SUSE 2020
    We Already Roll

    View Slide

  27. Copyright © SUSE 2020
    27
    We Already Roll
    SLE 15 GA - 3517 source packages
    SLE 15 SP1 - 1163 source packages
    SLE 15 SP2 - 1628 source packages
    SLE 15 Updates - 7455 update sources (updating 1296 packages)
    SLE 15 SP1 Updates - 2699 update sources (updating 482 packages)
    SLE 15 SP2 Updates - 835 update sources (updating 252 packages)
    PackageHub (SLE 15) – 8826 source packages, 1099 update sources
    PackageHub (SLE 15 SP1) – 8179 source packages, 1145 update sources
    PackageHub (SLE 15 SP2) – 10135 source packages, 237 update sources

    View Slide

  28. Copyright © SUSE 2020
    28
    Version Updates in Service Packs
    SLE 15 GA – 2579 RPM Version Changes (compared to 12 SP3)
    SLE 15 SP1 – 2095 RPM Version Changes (compared to GA)
    SLE 15 SP2 – 2855 RPM Version Changes (compared to SP1)
    Source: http://xcdchk.suse.de

    View Slide

  29. Copyright © SUSE 2020
    29
    Version Updates Occur in Update Repos

    View Slide

  30. Copyright © SUSE 2020
    30
    We Don’t Do It Perfectly (yet)
    jsc#PM-1876 “SLES parameter changes happen from time to time and
    having an impact to certain application”
    jsc#PM-2220 “Usability issue with installation of packages from package
    hub”
    jsc#PM-2216 Patch-level “Tagging” for (SUSE) MicroOS
    jsc#PM-2194 “document lifecycles and support of various components in
    SLE better”

    View Slide

  31. 31 Copyright © SUSE 2020
    Engineering
    Assumptions &
    Pitfalls

    View Slide

  32. Copyright © SUSE 2020
    32
    Stable Base + Rolling Layer
    “Customers want new stuff? Easy! Just
    keep the base stable and release faster
    stuff on top”

    View Slide

  33. Copyright © SUSE 2020
    33
    This is Theseus’ Paradox

    View Slide

  34. Copyright © SUSE 2020
    34
    Case Study – GregKH’s Tumbleweed (pre 2014)
    Tumbleweed was originally started by Greg Kroah-Hartman in 2010
    Rolling base atop regular openSUSE releases
    Focus on Kernel, KDE, GNOME and some desktop Apps
    Would overwrite/supersede packages from regular release
    ”Reset-to-zero” every regular release

    View Slide

  35. Copyright © SUSE 2020
    35
    Case Study – GregKH’s Tumbleweed (pre 2014)
    “Partially Rolling” was painful for both users and engineers
    Constant breakage over the growing chasm between the ‘stable’ base
    and rolling top
    Ad-hoc tinkering/superseding of the ‘stable’ base stops it being stable
    “Reset-to-zero” rebase to a new stable base every 8 months was brutally
    disruptive for all users

    View Slide

  36. Copyright © SUSE 2020
    36
    Lessons Learned – Modern Tumbleweed
    Evolved out of efforts to stabilise openSUSE:Factory
    Build all packages together, rebuild dependency tree as new/updated
    packages added (leveraging OBS)
    Test all relevant use cases, focusing on the way users use them
    (leveraging openQA, LTP, and various release bots)
    Sustainable engineered and usable for it’s target audience for 6 years
    running

    View Slide

  37. Copyright © SUSE 2020
    37
    Case Study – Modules/Package Hub
    jsc#PM-2220 “Usability issue with installation of packages from package
    hub”
    Package Hub packages can require packages from ANY module
    Customers have no way of knowing WHICH module is required when
    `zypper install` fails with a “nothing provides XXXX” error

    View Slide

  38. Copyright © SUSE 2020
    38
    Containerisation/Sandboxing
    “Customers want new stuff? Easy! Just
    keep the base stable and release faster
    stuff in containers”

    View Slide

  39. Copyright © SUSE 2020
    39
    Case Study - AppImage
    Portable Software format, containing binaries and required libraries in an
    executable archive
    Promises “Linux apps that run anywhere”
    Used by various upstreams to distribute their own binaries

    View Slide

  40. Copyright © SUSE 2020
    40
    Case Study - AppImage
    Does not run everywhere

    View Slide

  41. Copyright © SUSE 2020
    41
    Case Study - FlatPak
    os-tree based packaging and distribution format
    Uses common ‘runtimes’ to aid portability
    Promises to work on every Linux distribution

    View Slide

  42. Copyright © SUSE 2020
    42
    Case Study - FlatPak

    View Slide

  43. Copyright © SUSE 2020
    43
    Case Study - FlatPak
    Works on every Linux distribution that packages FlatPak
    ‘Runtimes’ are just containerised distributions, some focused on a
    specific ecosystem (eg. gtk, Qt) others with as broad a scope as a full
    blown distribution
    Base system still needs to have flatpak and all of its dependencies and
    kernels/core libraries that provide all the functionality required

    View Slide

  44. Copyright © SUSE 2020
    44
    Master
    openSUSE Kubic
    Case Study - Kubernetes
    Kubelet
    Container Runtime
    Control Plane
    Node
    openSUSE Kubic
    Kubelet
    Container Runtime
    Containers
    Node
    openSUSE Kubic
    Kubelet
    Container Runtime
    Containers
    Node
    openSUSE Kubic
    Kubelet
    Container Runtime
    Containers

    View Slide

  45. Copyright © SUSE 2020
    45
    Case Study - Kubernetes
    Control Plane containers must contain binaries equal to or newer than
    kubelet versions
    kubelet & container runtime are traditional binaries on the host OS
    Control Plane containers must be updated & cluster reconfigured before
    kubelet can be updated
    kubelet and container runtime must be compatible versions

    View Slide

  46. Copyright © SUSE 2020
    46
    Lessons Learned – openSUSE Kubic
    Control Plane containers built as part of traditional Tumbleweed/Kubic snapshot
    New Containers, kubelet, and container runtimes released at the same time
    kubelet packaged for multi-versioning (/usr/bin/kubelet1.xx)
    All supported kubelet versions installed on all nodes
    user-space tooling enables correct kubelet binary for deployed cluster version

    View Slide

  47. Copyright © SUSE 2020
    47
    Lessons Learned – openSUSE Kubic
    Distribution neutral/system isolated containers is a myth
    Building & releasing containers in alignment with traditional RPM
    packages is essential
    Containers can impose ‘unfair’ dependencies on the host OS that
    traditional packaging cannot model or resolve

    View Slide

  48. 48 Copyright © SUSE 2020
    Engineering
    “Rolling First”

    View Slide

  49. Copyright © SUSE 2020
    49
    “Rolling First”
    “Rolling First” Engineering is the concept of building our products with the
    following assumptions ingrained the mindset & processes:
    Binary & library versions will change over the lifespan of the product
    Functionality will be added and removed over the lifespan of the product
    Essential functionality and usability must be ensured throughout

    View Slide

  50. Copyright © SUSE 2020
    50
    “Rolling First”

    View Slide

  51. Copyright © SUSE 2020
    51
    The Rolling Engineering Axiom
    “In order to move ANYTHING quickly,
    you need to be able to move
    EVERYTHING quickly”

    View Slide

  52. Copyright © SUSE 2020
    52
    Unsafe At Any Speed?
    Full speed ahead is not the only speed
    Tumbleweed has proven processes for releasing at the pace of
    upstream/contributions
    ”Rolling First” within SUSE needs to release at the pace of the
    market/customers

    View Slide

  53. Copyright © SUSE 2020
    53
    “Rolling First” - Benefits
    Ability to respond to customer demands more rapidly
    Ability to keep up with and more effectively contribute to our essential
    upstreams
    Ability to distribute work more evenly throughout the calendar year
    Reduce corrective work by teams dependent on other products
    Does not necessarily need to impact how we release to customers

    View Slide

  54. Copyright © SUSE 2020
    54
    “Rolling First” - Risks
    Not every customer wants to move at the same speed as every other
    Not every upstream is aligned with their related and dependent
    codebases
    We will never know every use case our customers have for our products,
    so we can never be sure our products are suitable for everything

    View Slide

  55. Copyright © SUSE 2020
    55
    Testing, Testing, Testing
    Essential functionality for customers must be captured, modeled, tested
    and used as a release gate continuously
    No chance that manual testing can keep up
    openQA and other CI/CD tools and processes provide a good start, but
    investment will be needed from PM, QE, and Engineering

    View Slide

  56. Copyright © SUSE 2020
    56
    Scoping, Scoping, Scoping
    openSUSE MicroOS and Kubic have shown that narrowing the scope of
    the OS and leveraging containers can mitigate some of the risks of this
    approach
    Containers can ‘drift’ ahead or behind the base OS for brief periods
    without impacting compatibility
    A more narrowly defined base OS has fewer changes, and fewer risks
    introduced by those changes

    View Slide

  57. Copyright © SUSE 2020
    57
    Core
    Next?
    openSUSE Tumbleweed
    Leap
    15.2
    SLE
    15 SP2
    Core
    15.2
    Leap/Jump
    15.3
    SLE
    15 SP3
    Core
    15.3

    View Slide

  58. Copyright © SUSE 2020
    58
    Leap/Jump
    Next?
    CAP
    Next?
    SES
    Next?
    SUMA
    Next?
    CaaSP
    Next?
    ?
    ?
    SLE
    Next?
    Core
    Next?
    openSUSE Tumbleweed
    Leap
    15.2
    SLE
    15 SP2
    Core
    15.2
    Leap/Jump
    15.3
    SLE
    15 SP3
    Core
    15.3

    View Slide

  59. Copyright © SUSE 2020
    59
    Leap/Jump
    Next?
    CAP
    Next?
    SES
    Next?
    SUMA
    Next?
    CaaSP
    Next?
    ?
    ?
    SLE
    Next?
    Core
    Next?
    openSUSE Tumbleweed
    Leap
    15.2
    SLE
    15 SP2
    Core
    15.2
    Leap/Jump
    15.3
    SLE
    15 SP3
    Core
    15.3

    View Slide

  60. 60 Copyright © SUSE 2020
    Releasing “Rolling
    First”

    View Slide

  61. Copyright © SUSE 2020
    61
    Changing Customer Expectations

    View Slide

  62. Copyright © SUSE 2020
    62
    Changing Customer Expectations
    We have a history best aligned with late majority/laggard innovators
    Our newer products & acquisitions and Track 2 initiatives give us an
    opportunity to target the earlier adopters and majority
    This can be done either by offering something rolling & faster in parallel
    to our current offerings, or by adapting our current offerings to be rolling
    & faster
    Customers must be convinced, but the facts are on this side

    View Slide

  63. Thank you.
    © 2020 SUSE LLC. All Rights Reserved. SUSE and the
    SUSE logo are registered trademarks of SUSE LLC in the
    United States and other countries. All third-party
    trademarks are the property of their respective owners.
    For more information, contact SUSE at:
    +1 800 796 3700 (U.S./Canada)
    +49 (0)911-740 53-0 (Worldwide)
    SUSE
    Maxfeldstrasse 5
    90409 Nuremberg
    www.suse.com

    View Slide