Rust.Tokyo 2021: How Rust triages regressions and prevents them from reaching to stable

Transcript

1. How Rust triages regressions and prevents them from reaching to stable Yuki Okushi
2. Yuki Okushi - Rust contributor - Work at Huawei - Working on diagnostics and some docs (Rustnomicon, rustc-dev-guide, ...) - Maintaining some crates (libc, actix-*, ...) - A member of prioritization WG
3. Rust’s release channels
4. Rust’s release channels - Rust has three release channels - Stable - Beta - Nightly - Usually, people use Stable, or Nightly (for unstable and/or latest changes) - So, when is Beta used?
5. What are the different types of regressions?
6. What are the different types of regressions? - Correctness changes (ICE, miscompilation, ...) - Performance changes (binary size, memory usage, ...) - Rustdoc-related - ...
7. Regression types - regression-from-stable-to-nightly: exists only on nightly - regression-from-stable-to-beta: exists on beta - regression-from-stable-to-stable: exists on stable😱 - We should definitely prevent this!
8. How to find regressions before releasing stable?
9. 1. Crater
10. What’s Crater? Find regressions by building or testing code on Beta or particular PR
11. How to find? 1. Run `cargo check/build/test` and some tools (Clippy, Rustdoc) to a number of crates available on crates.io and GitHub 2. Compare the results with the latest Stable (or the current master branch if it’s PR)
12. Example: Crater on PR - #82565: Re-stabilize `integer::BITS`
13. Crater result
14. Example case: #82565 - Found one crate’s build failed - Confirmed it was fixed on newer version - The team confirmed that the change was safe → The PR has been merged with less pain
15. Check “Compatibility Notes” out on relnotes!
16. Example: Crater on Beta #86597: Crater runs for 1.54
17. Example regression #86831: infallible residual could not convert error
18. 2. Performance monitoring
19. Rustc's compile-time issue - Rustc is slow to compile (as you know) - Well, don’t the team care about it? → Yes, there are some effort!
20. Performance monitoring - perf.rust-lang.org - Run benchmarks on all the merged PRs - Triage them every week - Run benchmarks on perf-sensitive PRs before merging
21. Performance monitoring
22. Performance triage
23. Performance triage It’s also mentioned on TWiR
24. Example: benchmarks on PR #86385: Use `AttrVec` for `Arm`, `FieldDef`, and `Variant`
25. Regression triage
26. Regression triage - We’ve been doing our best but we cannot reduce the number of regressions to 0. - What should we do if we find them? → This is where the Prioritization WG comes in!
27. What’s the prioritization working group? - Prioritize the regressions reported on GitHub issue tracker - Monitor critical ones not to slip out - Nominate some of them to t-compiler meetings if necessary - ...so that we do not ship regressions to stable mistakenly More details about WG can be found here
28. Prioritization? - rust-lang/rust has four priority levels - Assign them to regressions
29. Priority levels - Critical - A release blocker, must be resolved/discussed until next release (e.g. unsound issue) - High - Not a blocker but should be resolved/triaged ASAP (e.g. working -> broken regression) - Medium - Low
30. Prioritization workflow: prioritizing regressions
31. Put them on the t-compiler meeting agenda
32. Beta backport
33. How to contribute it?
34. How to contribute it? Prioritization WG welcomes everyone! - How to participate? Can I actually contribute? - Anyone can join the Zulip stream and say “Hi”! - See this blog post for more details
35. But it’s not the only way!
36. Other ways: “Cleanup Crew” - Prepare Minimal Complete and Verifiable Example (MCVE) - Bisect a commit or nightly that introduced a regression - Find related/duplicate issues if exists More details can be found here
37. Other ways: Use Rust - Everyone could find a regression by using Rust. - Report it to the issue tracker if you find it and have some time Early reporting allows contributors more (earlier) time to work on a fix 🚢
38. How to create a great bug report? - Fill out the issue template as possible - Try the latest Nightly if you’re on Stable or an old Nightly - MCVE would be helpful! - There’s a useful blog post by Felix S Klock II - Backtrace would also be helpful
39. Conclusion
40. Conclusion - Rust has several ways to prevent regressions - Crater, performance monitoring, prioritization WG - The discussion is open - Everyone can read a log and leave a comment - Prioritizing is one of good ways to get your feet wet at contributing
41. Thanks!