RustFest 2017 - Taking Rust To Production: Lessons Learned From The Habitat Project

The Rust language is more than ready for production–it excels at it! The Habitat project team has been using Rust for a year in public and an extra year in stealth. With over 40,000 lines of Rust code so far, we maintain multiple CLI applications, a cross-platform process supervisor, a gossip subsystem, and a micro service-oriented distributed build system.

In this talk we will cover some of the challenges that Rust is uniquely qualified to address including static compilation, inlining dependencies, using clap for CLI parsing, pragmatically using native C libraries, maintaining a common codebase for Linux, Mac, and Windows, and on-boarding new Rustaceans. Have no fear: there has never been a better time to use Rust for your next service, application, or tool!

https://github.com/fnichol/talk-rustfest2017-habitat

Transcript

1. Taking Rust to Production: Lessons Learned From the Habitat Project Fletcher Nichol RustFest 2017 April 30, 2017 Kyiv, Ukraine
2. Is Rust ready to be used in production? Yes.
3. Fletcher Nichol Twitter: GitHub: Rust Users: Works on Habitat. Likes Rust. And summertime. Also, drums. @fnichol @fnichol @fnichol
5. is a runtime system for your apps or other services provides realtime service configuration packages software with a deterministic build system loves your apps is primarily written in Rust Habitat
6. https://www.habitat.sh
7. Why Rust?
8. This question comes up...
9. The Supervisor Small memory footprint Fast and responsive Low level system calls Can't crash
10. Code reuse and sharing
11. Bonus: CLI tools
12. Habitat Components in Rust The Supervisor The hab CLI A distributed build system (Builder)
13. It's a growing codebase 01 > date 02 Fri 28 Apr 2017 10:48:21 EEST 03 04 > cd habitat/ 05 06 > find . -name '*.rs' -exec cat {} \; \ 07 | egrep -v '^\s*$' \ 08 | egrep -v '^\s*//' \ 09 | wc -l 10 78551
14. How Habitat Uses Rust
15. Use native libraries
16. External libraries libarchive Archive reading and streaming libsodium All crypto applications (minus client TLS) zeromq rumor propagation, server communication
17. Static compilation
18. Build a dynamic binary cargo build --release
19. Dynamic binary: file 01 > file target/x86_64-unknown-linux-gnu/release/hab 02 target/x86_64-unknown-linux-gnu/release/hab: ELF 64-bit LSB » 03 shared object, x86-64, version 1 (SYSV), » 04 dynamically linked, » 05 interpreter /lib64/ld-linux-x86-64.so.2, » 06 for GNU/Linux 2.6.32, » 07 not stripped
20. Dynamic binary: ldd 01 > ldd target/x86_64-unknown-linux-gnu/release/hab 02 linux-vdso.so.1 (0x00007ffd2c66b000) 03 libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f6d524c4000) 04 libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f6d52071000) 05 libsodium.so.18 => /lib/x86_64-linux-gnu/libsodium.so.18 (0x00007f6d51db9000) 06 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6d51118000) 07 /lib64/ld-linux-x86-64.so.2 (0x00007f6d52735000) 08 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6d50e1a000) 09 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6d50bff000) 10 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f6d509cc000) 11 libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f6d507bc000) 12 ...
21. musl musl, a new standard library to power a new generation of Linux-based devices. musl is lightweight, fast, simple, free, and strives to be correct in the sense of standards-conformance and safety. - https://www.musl-libc.org/
22. Build with musl: setup rustup target add x86_64-unknown-linux-musl # compile or install a musl package # containing library and headers
23. Build with musl cargo build --release --target=x86_64-unknown-linux-musl
24. Statically build hab program export LIBARCHIVE_LIB_DIR=/usr/local/lib export LIBARCHIVE_INCLUDE_DIR=/usr/local/include export LIBARCHIVE_LDFLAGS="-L/usr/local/lib -lz -llzma ..." export LIBARCHIVE_STATIC=true export OPENSSL_LIB_DIR=/usr/local/lib export OPENSSL_INCLUDE_DIR=/usr/local/include export OPENSSL_STATIC=true export SODIUM_LIB_DIR=/usr/local/lib export SODIUM_STATIC=true cargo build --release --target=x86_64-unknown-linux-musl
25. Static binary: no external dependencies! 01 > file target/x86_64-unknown-linux-musl/release/hab 02 target/x86_64-unknown-linux-musl/hab: ELF 64-bit LSB » 03 executable, x86-64, version 1 (GNU/Linux), » 04 statically linked, stripped 05 06 07 > ldd target/x86_64-unknown-linux-musl/release/hab 08 not a dynamic executable 09 10 11 > du -h target/x86_64-unknown-linux-musl/release/hab 12 7.6M target/x86_64-unknown-linux-musl/release/hab
26. The "Mono-Repo" (Many software components in one repository)
27. Habitat components Directory . ├── components │ ├── backline │ ├── builder-admin │ ├── builder-admin-proxy ... │ ├── butterfly │ ├── butterfly-test │ ├── common │ ├── core ... │ ├── ruby-client │ ├── studio │ ├── sup │ └── win-users
28. Share build outputs and lockfile
29. Cargo workspaces A workspace is a set of packages that will all share the same Cargo.lock and output directory. - The Rust Programming Language, 2nd Ed.
30. Cargo.toml [workspace] members = [ "components/builder-admin", "components/builder-api", "components/builder-core", ... "components/butterfly", "components/butterfly-test", "components/common", ... "components/hab", "components/http-client", "components/net", "components/sup", ]
31. components/hab/Cargo.toml 01 [package] 02 name = "hab" 03 version = "0.0.0" 04 build = "../build.rs" 05 workspace = "../../"
32. Common versioning
33. components/build.rs (1/2) 01 fn main() { 02 let version = match env::var("PLAN_VERSION") { 03 Ok(ver) => ver, 04 _ => read_version(), 05 }; 06 let mut f = File::create( 07 Path::new(&env::var("OUT_DIR").unwrap()) 08 .join("VERSION")).unwrap(); 09 f.write_all(version.trim().as_bytes()).unwrap(); 10 }
34. components/build.rs (2/2) 01 fn read_version() -> String { 02 let ver_file = Path::new( 03 &env::var("CARGO_MANIFEST_DIR").unwrap()) 04 .parent() 05 .unwrap() 06 .parent() 07 .unwrap() 08 .join("VERSION"); 09 let f = File::open(ver_file).unwrap(); 10 let mut reader = BufReader::new(f); 11 let mut ver = String::new(); 12 reader.read_line(&mut ver).unwrap(); 13 ver 14 }
35. components/hab/src/lib.rs pub const VERSION: &'static str = include_str!(concat!(env!("OUT_DIR"), "/VERSION"));'
36. Crate dependencies
37. components/sup/Cargo.toml [dependencies] ansi_term = "*" bitflags = "*" byteorder = "*" clap = { version = "*", features = [ "suggestions", "color", "unstable" ] } env_logger = "*" features = "*" glob = "*" habitat_butterfly = { path = "../butterfly" } habitat_common = { path = "../common" } ...
38. Dependency strategy: aggressive
41. Cross platform codebase
42. Cross platform tips Centralize conditional code into a few places Keep public API calls consistent, if possible Read the Rust codebase, it has great examples!
43. components/core/src/os/filesyst em/mod.rs 01 #[cfg(windows)] 02 mod windows; 03 04 #[cfg(windows)] 05 pub use self::windows::{chown, chmod, symlink}; 06 07 #[cfg(not(windows))] 08 mod linux; 09 10 #[cfg(not(windows))] 11 pub use self::linux::{chown, chmod, symlink};
44. components/core/src/os/filesyst em/linux.rs 01 use libc::{self, c_int, c_char, mode_t}; 02 03 pub use std::os::unix::fs::symlink; 04 use std::ffi::CString; 05 06 pub fn chown(path: &str, uid: u32, gid: u32) -> Result<c_int> { 07 // ... 08 } 09 10 pub fn chmod(path: &str, mode: u32) -> Result<c_int> { 11 // ... 12 }
45. components/core/src/os/filesyst em/windows.rs 01 // ... 02 03 pub fn chown(path: &str, uid: String, gid: String) 04 -> Result<c_int> { 05 path_exists(path) 06 } 07 08 pub fn chmod(path: &str, mode: u32)-> Result<c_int> { 09 path_exists(path) 10 } 11 12 pub fn symlink<P: AsRef<Path>, Q: AsRef<Path>>(src: P, dst: Q) 13 -> io::Result<()> { 14 unimplemented!(); 15 }
46. Onboarding team members
48. Onboarding tips Give new members time and space to learn, experiment, ask questions Pair on features Talk about architecture and code's "point of view" Talk about .unwrap() and .expect() Be patient--it gets really good if you put the time in!
49. Rust pain points
50. Slow compile times
51. Hint: Cargo workspaces
52. Cross platform code paths
53. Hint: Continuous Integration
54. Rust is too fast?
55. Hint: Think about synchronization points
56. Wrapping up
57. Is Rust ready to be used in production? Yes, you bet.
58. http://slack.habitat.sh/
59. Thank you!