Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rust.Tokyo 2021: How Rust triages regressions and prevents them from reaching to stable

Yuki Okushi
September 18, 2021

Rust.Tokyo 2021: How Rust triages regressions and prevents them from reaching to stable

Yuki Okushi

September 18, 2021
Tweet

Other Decks in Technology

Transcript

  1. How Rust triages regressions
    and prevents them from
    reaching to stable
    Yuki Okushi

    View Slide

  2. Yuki Okushi
    - Rust contributor
    - Work at Huawei
    - Working on diagnostics and some docs (Rustnomicon,
    rustc-dev-guide, ...)
    - Maintaining some crates (libc, actix-*, ...)
    - A member of prioritization WG

    View Slide

  3. Rust’s release channels

    View Slide

  4. Rust’s release channels
    - Rust has three release channels
    - Stable
    - Beta
    - Nightly
    - Usually, people use Stable, or Nightly (for unstable and/or
    latest changes)
    - So, when is Beta used?

    View Slide

  5. What are the different types of regressions?

    View Slide

  6. What are the different types of regressions?
    - Correctness changes (ICE,
    miscompilation, ...)
    - Performance changes (binary size,
    memory usage, ...)
    - Rustdoc-related
    - ...

    View Slide

  7. Regression types
    - regression-from-stable-to-nightly: exists only
    on nightly
    - regression-from-stable-to-beta: exists on beta
    -
    regression-from-stable-to-stable: exists on
    stable😱
    - We should definitely prevent this!

    View Slide

  8. How to find regressions
    before releasing stable?

    View Slide

  9. 1. Crater

    View Slide

  10. What’s Crater?
    Find regressions by building or testing code on Beta or
    particular PR

    View Slide

  11. How to find?
    1. Run `cargo check/build/test` and some tools (Clippy,
    Rustdoc) to a number of crates available on crates.io and
    GitHub
    2. Compare the results with the latest Stable (or the current
    master branch if it’s PR)

    View Slide

  12. Example: Crater on PR
    - #82565: Re-stabilize `integer::BITS`

    View Slide

  13. Crater result

    View Slide

  14. Example case: #82565
    - Found one crate’s build failed
    - Confirmed it was fixed on newer version
    - The team confirmed that the change was safe
    → The PR has been merged with less pain

    View Slide

  15. Check “Compatibility Notes” out on relnotes!

    View Slide

  16. Example: Crater on Beta
    #86597: Crater runs for 1.54

    View Slide

  17. Example regression
    #86831: infallible residual could not convert error

    View Slide

  18. 2. Performance monitoring

    View Slide

  19. Rustc's compile-time issue
    - Rustc is slow to compile (as you know)
    - Well, don’t the team care about it?
    → Yes, there are some effort!

    View Slide

  20. Performance monitoring
    - perf.rust-lang.org
    - Run benchmarks on all the merged PRs
    - Triage them every week
    - Run benchmarks on perf-sensitive PRs before merging

    View Slide

  21. Performance monitoring

    View Slide

  22. Performance triage

    View Slide

  23. Performance triage
    It’s also mentioned on TWiR

    View Slide

  24. Example: benchmarks on PR
    #86385: Use `AttrVec` for `Arm`, `FieldDef`, and `Variant`

    View Slide

  25. Regression triage

    View Slide

  26. Regression triage
    - We’ve been doing our best but we cannot reduce the
    number of regressions to 0.
    - What should we do if we find them?
    → This is where the Prioritization WG comes in!

    View Slide

  27. What’s the prioritization working group?
    - Prioritize the regressions reported on GitHub issue tracker
    - Monitor critical ones not to slip out
    - Nominate some of them to t-compiler meetings if
    necessary
    - ...so that we do not ship regressions to stable
    mistakenly
    More details about WG can be found here

    View Slide

  28. Prioritization?
    - rust-lang/rust has four
    priority levels
    - Assign them to
    regressions

    View Slide

  29. Priority levels
    - Critical
    - A release blocker, must be resolved/discussed until
    next release (e.g. unsound issue)
    - High
    - Not a blocker but should be resolved/triaged ASAP
    (e.g. working -> broken regression)
    - Medium
    - Low

    View Slide

  30. Prioritization workflow: prioritizing regressions

    View Slide

  31. Put them on the t-compiler meeting agenda

    View Slide

  32. Beta backport

    View Slide

  33. How to contribute it?

    View Slide

  34. How to contribute it?
    Prioritization WG welcomes everyone!
    - How to participate? Can I actually contribute?
    - Anyone can join the Zulip stream and say “Hi”!
    - See this blog post for more details

    View Slide

  35. But it’s not the only way!

    View Slide

  36. Other ways: “Cleanup Crew”
    - Prepare Minimal Complete and Verifiable Example
    (MCVE)
    - Bisect a commit or nightly that introduced a regression
    - Find related/duplicate issues if exists
    More details can be found here

    View Slide

  37. Other ways: Use Rust
    - Everyone could find a regression by using Rust.
    - Report it to the issue tracker if you find it and have some
    time
    Early reporting allows contributors more (earlier) time to
    work on a fix 🚢

    View Slide

  38. How to create a great bug report?
    - Fill out the issue template as possible
    - Try the latest Nightly if you’re on Stable or an old Nightly
    - MCVE would be helpful!
    - There’s a useful blog post by Felix S Klock II
    - Backtrace would also be helpful

    View Slide

  39. Conclusion

    View Slide

  40. Conclusion
    - Rust has several ways to prevent regressions
    - Crater, performance monitoring, prioritization WG
    - The discussion is open
    - Everyone can read a log and leave a comment
    - Prioritizing is one of good ways to get your feet wet at
    contributing

    View Slide

  41. Thanks!

    View Slide