A peek behind the curtain: Managing the Kubernetes Contributor Community

The Kubernetes community is a vibrant beacon in open source. It takes a village to enable a city of contributors doing what they do best. There are a lot of fun stories and lessons to be shared from helping out the community. One lesson is taken straight from the Kubernetes project itself: declarative config management. Most aspects of the community are managed using declarative configs. Adding a new SIG, GitHub org member, and even Slack channel, involves updating and PRing a change into one of the many Kubernetes repos. Adopting this methodology provides the community the means to self-manage itself. Join us as we journey through the many bits of community automation and weigh the merits of automating every aspect of our community.

https://www.socallinuxexpo.org/scale/18x/presentations/peek-behind-curtain-managing-kubernetes-contributor-community

Transcript

1. Managing the Kubernetes
   Contributor Community
   A peek behind the curtain
   Bob Killen
   @mrbobbytables
   Jeff Sica
   @jeefy
   

   View Slide

2. View Slide

3. $ whoami - Jeff
   Jeffrey Sica
   jeef111x@gmail.com
   Senior Software Engineer @
   CNCF Ambassador
   Github: @jeefy
   Twitter: @jeefy
   

   View Slide

4. $ whoami - Bob
   Bob Killen
   bob.killen@linux.com
   Senior Research Cloud Administrator
   CNCF Ambassador
   Github: @mrbobbytables
   Twitter: @mrbobbytables
   

   View Slide

5. tl;dr Kubernetes
   ● Open Source vendor-neutral container orchestration system initially created
   by Google in 2014
   ● Donated as the inception project to the CNCF (Cloud Native Computing
   Foundation) in 2015
   ● Grown to become the standard for container orchestration
   Image Source
   @jeefy
   @mrbobbytables
   

   View Slide

6. What is the Kubernetes Community
   #1 OSS project by developer activity*
   #2 project by Pull Requests*
   Source: devstats
   Community Stats (2020-02-21)
   Contributors 42,000+
   Org Members 1,225
   Repos 209
   Pull Requests 148,000~
   Slack 92,000+
   * As of 2020-02-25 - Ref: CNCF Velocity Report
   @jeefy
   @mrbobbytables
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. @jeefy
    @mrbobbytables
    Distribution is better than centralization
    Community over product
    or company
    Automation over process
    Inclusive is better
    than exclusive
    Evolution is better
    than stagnation
    Source: http://git.k8s.io/community/values.md
    

    View Slide

12. The first contribution...
    @jeefy
    @mrbobbytables
    

    View Slide

13. The first contribution...
    @jeefy
    @mrbobbytables
    First interaction will be
    with our bots
    

    View Slide

14. A word on CI...Prow
    ● Prow
    ● Supports three types of jobs:
    ○ Periodics
    ○ Pre-submits
    ○ Post-submits
    ● Project CI is defined within one
    repo: test-infra
    ● Performs tests and merges PRs
    based on labels and rules
    

    View Slide

15. A word on CI...ProwJob
    presubmits:
    kubernetes/org:
    - name: pull-org-verify-all
    always_run: true
    decorate: true
    labels:
    preset-service-account: "true"
    preset-bazel-scratch-dir: "true"
    spec:
    Containers:
    - image: launcher.gcr.io/google/bazel:0.29.1
    command:
    - ./hack/verify-all.sh
    annotations:
    testgrid-num-columns-recent: '30'
    testgrid-create-test-group: 'true'
    

    View Slide

16. A word on CI...Tests
    

    View Slide

17. The first contribution...labels
    @jeefy
    @mrbobbytables
    ● Provide verbose messages regarding PR
    ● Check state of PR
    ○ Check CLA
    ○ Check kind label (bug, feature, etc)
    ○ Check priority
    ○ Check commit message
    ○ Check for release notes in PR template
    ○ Applies group (SIG) labels based on files being
    updated (more on this later…)
    

    View Slide

18. ...but first! The CLA
    

    View Slide

19. ...but first! The CLA
    

    View Slide

20. The first contribution...assigning reviewers
    @jeefy
    @mrbobbytables
    ● Prow Plugin: Blunderbuss
    ● Reviewers are selected from OWNERS files
    ● Inspired by Chromium’s OWNERS files / GitHub’s
    CODEOWNERS files
    ● Each directory can contain an independent
    OWNERS file
    ○ If not found in current directory, it walks backwards up
    the directory tree till one is found
    ● Root of repo can contain a aliases to groups of
    contributors in an OWNERS_ALIASES file
    approvers:
    - alice
    - bob
    reviewers:
    - alice
    - carl
    - sig-foo # alias
    labels:
    - sig/foo
    OWNERS
    

    View Slide

21. The first contribution...assigning reviewers
    @jeefy
    @mrbobbytables
    ● Prow Plugin: Blunderbuss
    ● Reviewers are selected from OWNERS files
    ● Inspired by Chromium’s OWNERS files / GitHub’s
    CODEOWNERS files
    ● Each directory can contain an independent
    OWNERS file
    ○ If not found in current directory, it walks backwards up
    the directory tree till one is found
    ● Root of repo can contain a aliases to groups of
    contributors in an OWNERS_ALIASES file
    

    View Slide

22. All BEFORE tests are run
    

    View Slide

23. The first contribution...ok-to-test
    @jeefy
    @mrbobbytables
    ● /ok-to-test allows CI to be run for PRs
    proposed from non Kubernetes Org members.
    ● Any Org member may comment with
    /ok-to-test
    ● Commonly applied by reviewers after first
    pass a PR
    

    View Slide

24. The first contribution...ok-to-test
    ● /ok-to-test allows CI to be run for PRs
    proposed from non Kubernetes Org members.
    ● Any Org member may comment with
    /ok-to-test
    ● Commonly applied by reviewers after first
    pass a PR
    @jeefy
    @mrbobbytables
    

    View Slide

25. The first contribution...tests
    ● Testing Kubernetes is expensive
    ○ 1.5 - 2 hours to complete the full test suite
    ● Retesting of just failed jobs
    ● Flakes are sadly still common :(
    @jeefy
    @mrbobbytables
    

    View Slide

26. The first contribution...lgtm
    ● Prow Plugin: LGTM
    ● Any org Member may apply the lgtm label
    ● Commonly applied by reviewers after
    comments or by those they loop in
    ● Signals the PR should be good for final review
    from an approver
    @jeefy
    @mrbobbytables
    

    View Slide

27. The first contribution...approve
    approvers:
    - alice
    - bob
    reviewers:
    - alice
    - carl
    - sig-foo # alias
    labels:
    - sig/foo
    OWNERS
    @jeefy
    @mrbobbytables
    ● Prow Plugin: Approvers
    ● Only approvers listed in an OWNERS
    file may approve
    

    View Slide

28. The first contribution...approve
    ● Prow Plugin: Approvers
    ● Only approvers listed in an OWNERS
    file may approve
    ● Requires /approve from each area
    the PR touches or a higher level
    approver.
    @jeefy
    @mrbobbytables
    

    View Slide

29. The first contribution...approve
    ● Prow Plugin: Approvers
    ● Only approvers listed in an OWNERS
    file may approve
    ● Requires /approve from each area
    the PR touches or a higher level
    approver.
    @jeefy
    @mrbobbytables
    PR will now be merged \o/
    

    View Slide

30. Finding Your Place in
    the Community
    

    View Slide

31. Community Groups
    Special Interest Group
    Primary organizational
    unit of the Kubernetes
    Project.
    Code developed by the
    project must be owned
    by a SIG.
    Working Group
    Short lived groups to
    tackle cross-cutting SIG
    efforts.
    Code is owned by one
    or more of the
    sponsoring SIGs.
    User Group
    Provide a means for
    end users to
    collaborate along with a
    unifying voice to drive
    specific features.
    Cannot own code.
    Committee
    Handle sensitive topics
    (security, Code of
    Conduct etc)
    No open membership;
    Members are elected or
    appointed
    @jeefy
    @mrbobbytables
    

    View Slide

32. Release
    Contributor
    Experience
    PM
    Docs Testing
    API
    Machinery
    CLI UI
    Multicluster Windows
    Auth
    Apps Autoscaling
    Cluster
    Lifecycle
    Instrumentation
    Network Node
    Scalability
    Scheduling
    Service
    Catalog
    Storage
    Resource
    Management
    Steering
    Project
    Horizontal
    Vertical
    Architecture
    Code of
    Conduct
    Product
    Security
    Big Data
    Cloud
    Provider
    Component
    Standard
    IoT Edge
    K8s infra
    Machine
    Learning
    Multitenancy Policy
    Security
    Audit
    LTS
    Apply
    Usability
    Applications Resource Management Infrastructure
    Working
    group
    SIG
    Committee
    User group
    VMware
    Data
    Protection
    Community Group Structure
    

    View Slide

33. The Community Repo
    ● Community groups are managed in
    the kubernetes/community repo.
    ● Each group has a directory that
    contains information about the
    group itself
    ○ Charter
    ○ README
    @jeefy
    @mrbobbytables
    

    View Slide

34. Source of truth...sigs.yaml
    ● Metadata regarding the group is
    stored in sigs.yaml
    ● Human friendly version:
    sig-list.md
    ● Community Group READMEs are
    rendered using go template using
    the information from sigs.yaml
    

    View Slide

35. Getting involved...Zoom
    ● Every SIG/WG/UG has regularly
    scheduled meetings
    ● All meetings are uploaded to the
    Kubernetes Community YouTube
    Channel
    ○ In process of being automated via
    splain.io Zoom -> Youtube bridge
    @jeefy
    @mrbobbytables
    

    View Slide

36. Getting involved...Zoom
    ● Every SIG/WG/UG has regularly
    scheduled meetings
    ● All meetings are uploaded to the
    Kubernetes Community YouTube
    Channel
    ○ In process of being automated via
    splain.io Zoom -> Youtube bridge
    @jeefy
    @mrbobbytables
    ...sometimes we get a little slap happy
    

    View Slide

37. The ladder
    @jeefy
    @mrbobbytables
    Subproject Owner - Set priorities and approve proposals for subproject
    - Responsibility and leadership for entire repository/directory
    Approver - Approve contributions for acceptance
    - Highly experienced reviewer and contributor in subproject
    Reviewer - History of reviewing; reviews frequently
    - Authorship in subproject
    Member - Active contributor to the project
    - Sponsored by two Reviewers
    Non-member Contributors
    

    View Slide

38. Becoming an Org Member
    It’s about building trust...
    @jeefy
    @mrbobbytables
    

    View Slide

39. Becoming an Org Member
    ● Community Membership Requirements
    ● Be active within the community
    ● Find two sponsors (a reviewer or
    approver from different companies)
    ● File a GitHub Issue
    That’s It!
    @jeefy
    @mrbobbytables
    

    View Slide

40. Peribolos
    ● Prow Plugin: Peribolos
    ● GitOps for GitHub management
    ● Defines base org settings
    ● Org Membership
    ● GitHub Teams
    name: Kubernetes
    description: Kubernetes
    default_repository_permission: read
    has_organization_projects: true
    has_repository_projects: true
    members_can_create_repositories: false
    billing_email: github@kubernetes.io
    admins:
    - cblecker
    ...
    members:
    - 27149chen
    ...
    teams:
    metrics-admins:
    description: Admin access to the metrics repo
    members:
    - DirectXMan12
    ...
    privacy: closed
    @jeefy
    @mrbobbytables
    

    View Slide

41. Peribolos
    Team management can be delegated...
    @jeefy
    @mrbobbytables
    teams:
    sig-architecture-api-reviews:
    description: ...
    members:
    - bgrant0607
    - liggitt
    - smarterclayton
    privacy: closed
    sig-architecture-bugs:
    description: ...
    members:
    - bgrant0607
    privacy: closed
    ...
    

    View Slide

42. Becoming an Org Member
    @jeefy
    @mrbobbytables
    

    View Slide

43. Growing your role
    

    View Slide

44. Slack
    ● 92,000+ Users
    ● 36 regions represented
    @jeefy
    @mrbobbytables
    ● 380+ Channels
    ● 700+ Custom Emojis
    

    View Slide

45. Slack
    ● 92,000+ Users
    ● 36 regions represented
    @jeefy
    @mrbobbytables
    ● 380+ Channels
    ● 700+ Custom Emojis
    ...We do like our emojis
    

    View Slide

46. Slack...complications
    ● No audit trail without contacting slack
    ● Users cannot block other users
    ● Handles are free-text and can have duplicates
    ● Have to use 3rd party app for “open” sign-up
    ● Undocumented or inconsistent APIs
    @jeefy
    @mrbobbytables
    ..BUT it’s the standard because folks use it for $dayjob
    

    View Slide

47. Making Slack Safe for Open Communities
    ● Slack-Infra Tools
    ○ Slack-welcomer - Sends a welcome message to every new user who
    joins Slack
    ○ Slack-event-log - Global event logging for Slack
    ○ Slack-report-message - Enables Slack users to report messages;
    sending them to a channel
    ○ Slack-moderator - Allows an Admin or Owner to inactive a user and
    delete their content.
    ○ Tempelis - GitOps for Slack channels and User Groups
    @jeefy
    @mrbobbytables
    Katharine Berry
    

    View Slide

48. Slack Reporter
    @jeefy
    @mrbobbytables
    

    View Slide

49. Slack Reporter
    @jeefy
    @mrbobbytables
    ¯\_(ツ)_/¯
    

    View Slide

50. Tempelis
    ● Manages channels and usergroups
    ○ Create / Update / Archive
    ● Management can be delegated by use
    of restrictions
    @jeefy
    @mrbobbytables
    restrictions:
    ...
    - path: "sig-docs/*.yaml"
    channels:
    - "^kubernetes-docs-[a-z]{2}$"
    - path: "sig-release/*.yaml"
    channels:
    - "^sig-release$"
    - "^release-"
    usergroups:
    - "^release-"
    ...
    

    View Slide

51. Tempelis
    @jeefy
    @mrbobbytables
    restrictions:
    ...
    - path: "sig-docs/*.yaml"
    channels:
    - "^kubernetes-docs-[a-z]{2}$"
    - path: "sig-release/*.yaml"
    channels:
    - "^sig-release$"
    - "^release-"
    usergroups:
    - "^release-"
    ...
    ● Manages channels and usergroups
    ○ Create / Update / Archive
    ● Management can be delegated by use
    of restrictions
    

    View Slide

52. Tempelis Channel Delegation
    @jeefy
    @mrbobbytables
    

    View Slide

53. Tempelis User Groups
    @jeefy
    @mrbobbytables
    users:
    alejandrox1: U6AS37R50
    aleksandra-malinowska: U357LUPHS
    bubblemelon: U7K9C643G
    calebamiles: U1ZDD4CUR
    castrojo: U1W1Q6PRQ
    ...
    sumitranr: UCQN13L9H
    tpepper: U6UB5V4TX
    Tunde: UAY977ENN
    

    View Slide

54. Tempelis User Groups
    @jeefy
    @mrbobbytables
    restrictions:
    ...
    - path: "sig-release/*.yaml"
    channels:
    - "^sig-release$"
    - "^release-"
    usergroups:
    - "^release-"
    ...
    

    View Slide

55. K8s.io: Google Group Membership
    ● Project: k8s.io/groups
    ● GitOps for Gsuite Google
    Group Management
    ○ Manages 70 *@kubernetes.io
    accounts
    ○ NOT used for community group
    accounts (SIG/WG/UG)
    ● Token is secured in repo via
    git-crypt
    groups:
    ...
    - email-id: community@kubernetes.io
    name: community
    description: |-
    settings:
    WhoCanPostMessage: "ANYONE_CAN_POST"
    ReconcileMembers: "true"
    owners:
    - ihor@cncf.io
    - jorgec@vmware.com
    - killen.bob@gmail.com
    - parispittman@google.com
    managers:
    - pal.nabarun95@gmail.com
    members:
    - dgiles@linuxfoundation.com
    - jberkus@redhat.com
    

    View Slide

56. K8s.io: Domains
    ● Project: k8s.io/dns
    ● Use GitHub’s ocotoDNS to manage all
    Kubernetes owned domains
    ● All domains / url rewrites are managed via source control
    

    View Slide

57. Contributor Summits
    8 Events since 2014
    Currently spanning EU, APAC, and NA
    Contributor-focused content
    New Contributor Workshops for onboarding
    Pre-summit socials
    “When you’re Kubernetes, you’re family” - Ancient Proverb
    

    View Slide

58. Steering and Elections
    ● Steering Committee oversees project governance
    and defines the overall project values and structure
    ○ NOT the technical direction of the project
    ○ 7 Members
    ○ 2 year term staggered by 4 / 3 split
    ● Voting Requirements
    ○ 50 DevStats recorded contributions within the past year
    ○ Exceptions allowed and vetted by election committee
    

    View Slide

59. DevStats
    ● Devstats project
    ● https://devstats.cncf.io/
    ● Toolset to visualize GitHub archives
    (GitHub events) using Grafana
    dashboards
    ● Data available for every CNCF
    project
    ● Exportable for further analysis
    

    View Slide

60. View Slide

61. Automation is meant to help people.
    

    View Slide

62. Thank you!
    

    View Slide