Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rust.Tokyo 2021: How Rust triages regressions and prevents them from reaching to stable

Yuki Okushi
September 18, 2021

Rust.Tokyo 2021: How Rust triages regressions and prevents them from reaching to stable

Yuki Okushi

September 18, 2021
Tweet

Other Decks in Technology

Transcript

  1. How Rust triages regressions
    and prevents them from
    reaching to stable
    Yuki Okushi

    View full-size slide

  2. Yuki Okushi
    - Rust contributor
    - Work at Huawei
    - Working on diagnostics and some docs (Rustnomicon,
    rustc-dev-guide, ...)
    - Maintaining some crates (libc, actix-*, ...)
    - A member of prioritization WG

    View full-size slide

  3. Rust’s release channels

    View full-size slide

  4. Rust’s release channels
    - Rust has three release channels
    - Stable
    - Beta
    - Nightly
    - Usually, people use Stable, or Nightly (for unstable and/or
    latest changes)
    - So, when is Beta used?

    View full-size slide

  5. What are the different types of regressions?

    View full-size slide

  6. What are the different types of regressions?
    - Correctness changes (ICE,
    miscompilation, ...)
    - Performance changes (binary size,
    memory usage, ...)
    - Rustdoc-related
    - ...

    View full-size slide

  7. Regression types
    - regression-from-stable-to-nightly: exists only
    on nightly
    - regression-from-stable-to-beta: exists on beta
    -
    regression-from-stable-to-stable: exists on
    stable😱
    - We should definitely prevent this!

    View full-size slide

  8. How to find regressions
    before releasing stable?

    View full-size slide

  9. What’s Crater?
    Find regressions by building or testing code on Beta or
    particular PR

    View full-size slide

  10. How to find?
    1. Run `cargo check/build/test` and some tools (Clippy,
    Rustdoc) to a number of crates available on crates.io and
    GitHub
    2. Compare the results with the latest Stable (or the current
    master branch if it’s PR)

    View full-size slide

  11. Example: Crater on PR
    - #82565: Re-stabilize `integer::BITS`

    View full-size slide

  12. Crater result

    View full-size slide

  13. Example case: #82565
    - Found one crate’s build failed
    - Confirmed it was fixed on newer version
    - The team confirmed that the change was safe
    → The PR has been merged with less pain

    View full-size slide

  14. Check “Compatibility Notes” out on relnotes!

    View full-size slide

  15. Example: Crater on Beta
    #86597: Crater runs for 1.54

    View full-size slide

  16. Example regression
    #86831: infallible residual could not convert error

    View full-size slide

  17. 2. Performance monitoring

    View full-size slide

  18. Rustc's compile-time issue
    - Rustc is slow to compile (as you know)
    - Well, don’t the team care about it?
    → Yes, there are some effort!

    View full-size slide

  19. Performance monitoring
    - perf.rust-lang.org
    - Run benchmarks on all the merged PRs
    - Triage them every week
    - Run benchmarks on perf-sensitive PRs before merging

    View full-size slide

  20. Performance monitoring

    View full-size slide

  21. Performance triage

    View full-size slide

  22. Performance triage
    It’s also mentioned on TWiR

    View full-size slide

  23. Example: benchmarks on PR
    #86385: Use `AttrVec` for `Arm`, `FieldDef`, and `Variant`

    View full-size slide

  24. Regression triage

    View full-size slide

  25. Regression triage
    - We’ve been doing our best but we cannot reduce the
    number of regressions to 0.
    - What should we do if we find them?
    → This is where the Prioritization WG comes in!

    View full-size slide

  26. What’s the prioritization working group?
    - Prioritize the regressions reported on GitHub issue tracker
    - Monitor critical ones not to slip out
    - Nominate some of them to t-compiler meetings if
    necessary
    - ...so that we do not ship regressions to stable
    mistakenly
    More details about WG can be found here

    View full-size slide

  27. Prioritization?
    - rust-lang/rust has four
    priority levels
    - Assign them to
    regressions

    View full-size slide

  28. Priority levels
    - Critical
    - A release blocker, must be resolved/discussed until
    next release (e.g. unsound issue)
    - High
    - Not a blocker but should be resolved/triaged ASAP
    (e.g. working -> broken regression)
    - Medium
    - Low

    View full-size slide

  29. Prioritization workflow: prioritizing regressions

    View full-size slide

  30. Put them on the t-compiler meeting agenda

    View full-size slide

  31. Beta backport

    View full-size slide

  32. How to contribute it?

    View full-size slide

  33. How to contribute it?
    Prioritization WG welcomes everyone!
    - How to participate? Can I actually contribute?
    - Anyone can join the Zulip stream and say “Hi”!
    - See this blog post for more details

    View full-size slide

  34. But it’s not the only way!

    View full-size slide

  35. Other ways: “Cleanup Crew”
    - Prepare Minimal Complete and Verifiable Example
    (MCVE)
    - Bisect a commit or nightly that introduced a regression
    - Find related/duplicate issues if exists
    More details can be found here

    View full-size slide

  36. Other ways: Use Rust
    - Everyone could find a regression by using Rust.
    - Report it to the issue tracker if you find it and have some
    time
    Early reporting allows contributors more (earlier) time to
    work on a fix 🚢

    View full-size slide

  37. How to create a great bug report?
    - Fill out the issue template as possible
    - Try the latest Nightly if you’re on Stable or an old Nightly
    - MCVE would be helpful!
    - There’s a useful blog post by Felix S Klock II
    - Backtrace would also be helpful

    View full-size slide

  38. Conclusion
    - Rust has several ways to prevent regressions
    - Crater, performance monitoring, prioritization WG
    - The discussion is open
    - Everyone can read a log and leave a comment
    - Prioritizing is one of good ways to get your feet wet at
    contributing

    View full-size slide