Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RustFest 2017 - Taking Rust To Production: Lessons Learned From The Habitat Project

RustFest 2017 - Taking Rust To Production: Lessons Learned From The Habitat Project

The Rust language is more than ready for production–it excels at it! The Habitat project team has been using Rust for a year in public and an extra year in stealth. With over 40,000 lines of Rust code so far, we maintain multiple CLI applications, a cross-platform process supervisor, a gossip subsystem, and a micro service-oriented distributed build system.

In this talk we will cover some of the challenges that Rust is uniquely qualified to address including static compilation, inlining dependencies, using clap for CLI parsing, pragmatically using native C libraries, maintaining a common codebase for Linux, Mac, and Windows, and on-boarding new Rustaceans. Have no fear: there has never been a better time to use Rust for your next service, application, or tool!

https://github.com/fnichol/talk-rustfest2017-habitat

9891e8299426fb9b6e361b84b3155a2d?s=128

Fletcher Nichol

June 14, 2017
Tweet

More Decks by Fletcher Nichol

Other Decks in Technology

Transcript

  1. Taking Rust to Production: Lessons Learned From the Habitat Project

    Fletcher Nichol RustFest 2017 April 30, 2017 Kyiv, Ukraine
  2. Is Rust ready to be used in production? Yes.

  3. Fletcher Nichol Twitter: GitHub: Rust Users: Works on Habitat. Likes

    Rust. And summertime. Also, drums. @fnichol @fnichol @fnichol
  4. None
  5. is a runtime system for your apps or other services

    provides realtime service configuration packages software with a deterministic build system loves your apps is primarily written in Rust Habitat
  6. https://www.habitat.sh

  7. Why Rust?

  8. This question comes up...

  9. The Supervisor Small memory footprint Fast and responsive Low level

    system calls Can't crash
  10. Code reuse and sharing

  11. Bonus: CLI tools

  12. Habitat Components in Rust The Supervisor The hab CLI A

    distributed build system (Builder)
  13. It's a growing codebase 01 > date 02 Fri 28

    Apr 2017 10:48:21 EEST 03 04 > cd habitat/ 05 06 > find . -name '*.rs' -exec cat {} \; \ 07 | egrep -v '^\s*$' \ 08 | egrep -v '^\s*//' \ 09 | wc -l 10 78551
  14. How Habitat Uses Rust

  15. Use native libraries

  16. External libraries libarchive Archive reading and streaming libsodium All crypto

    applications (minus client TLS) zeromq rumor propagation, server communication
  17. Static compilation

  18. Build a dynamic binary cargo build --release

  19. Dynamic binary: file 01 > file target/x86_64-unknown-linux-gnu/release/hab 02 target/x86_64-unknown-linux-gnu/release/hab: ELF

    64-bit LSB » 03 shared object, x86-64, version 1 (SYSV), » 04 dynamically linked, » 05 interpreter /lib64/ld-linux-x86-64.so.2, » 06 for GNU/Linux 2.6.32, » 07 not stripped
  20. Dynamic binary: ldd 01 > ldd target/x86_64-unknown-linux-gnu/release/hab 02 linux-vdso.so.1 (0x00007ffd2c66b000)

    03 libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f6d524c4000) 04 libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f6d52071000) 05 libsodium.so.18 => /lib/x86_64-linux-gnu/libsodium.so.18 (0x00007f6d51db9000) 06 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6d51118000) 07 /lib64/ld-linux-x86-64.so.2 (0x00007f6d52735000) 08 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6d50e1a000) 09 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6d50bff000) 10 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f6d509cc000) 11 libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f6d507bc000) 12 ...
  21. musl musl, a new standard library to power a new

    generation of Linux-based devices. musl is lightweight, fast, simple, free, and strives to be correct in the sense of standards-conformance and safety. - https://www.musl-libc.org/
  22. Build with musl: setup rustup target add x86_64-unknown-linux-musl # compile

    or install a musl package # containing library and headers
  23. Build with musl cargo build --release --target=x86_64-unknown-linux-musl

  24. Statically build hab program export LIBARCHIVE_LIB_DIR=/usr/local/lib export LIBARCHIVE_INCLUDE_DIR=/usr/local/include export LIBARCHIVE_LDFLAGS="-L/usr/local/lib

    -lz -llzma ..." export LIBARCHIVE_STATIC=true export OPENSSL_LIB_DIR=/usr/local/lib export OPENSSL_INCLUDE_DIR=/usr/local/include export OPENSSL_STATIC=true export SODIUM_LIB_DIR=/usr/local/lib export SODIUM_STATIC=true cargo build --release --target=x86_64-unknown-linux-musl
  25. Static binary: no external dependencies! 01 > file target/x86_64-unknown-linux-musl/release/hab 02

    target/x86_64-unknown-linux-musl/hab: ELF 64-bit LSB » 03 executable, x86-64, version 1 (GNU/Linux), » 04 statically linked, stripped 05 06 07 > ldd target/x86_64-unknown-linux-musl/release/hab 08 not a dynamic executable 09 10 11 > du -h target/x86_64-unknown-linux-musl/release/hab 12 7.6M target/x86_64-unknown-linux-musl/release/hab
  26. The "Mono-Repo" (Many software components in one repository)

  27. Habitat components Directory . ├── components │ ├── backline │

    ├── builder-admin │ ├── builder-admin-proxy ... │ ├── butterfly │ ├── butterfly-test │ ├── common │ ├── core ... │ ├── ruby-client │ ├── studio │ ├── sup │ └── win-users
  28. Share build outputs and lockfile

  29. Cargo workspaces A workspace is a set of packages that

    will all share the same Cargo.lock and output directory. - The Rust Programming Language, 2nd Ed.
  30. Cargo.toml [workspace] members = [ "components/builder-admin", "components/builder-api", "components/builder-core", ... "components/butterfly",

    "components/butterfly-test", "components/common", ... "components/hab", "components/http-client", "components/net", "components/sup", ]
  31. components/hab/Cargo.toml 01 [package] 02 name = "hab" 03 version =

    "0.0.0" 04 build = "../build.rs" 05 workspace = "../../"
  32. Common versioning

  33. components/build.rs (1/2) 01 fn main() { 02 let version =

    match env::var("PLAN_VERSION") { 03 Ok(ver) => ver, 04 _ => read_version(), 05 }; 06 let mut f = File::create( 07 Path::new(&env::var("OUT_DIR").unwrap()) 08 .join("VERSION")).unwrap(); 09 f.write_all(version.trim().as_bytes()).unwrap(); 10 }
  34. components/build.rs (2/2) 01 fn read_version() -> String { 02 let

    ver_file = Path::new( 03 &env::var("CARGO_MANIFEST_DIR").unwrap()) 04 .parent() 05 .unwrap() 06 .parent() 07 .unwrap() 08 .join("VERSION"); 09 let f = File::open(ver_file).unwrap(); 10 let mut reader = BufReader::new(f); 11 let mut ver = String::new(); 12 reader.read_line(&mut ver).unwrap(); 13 ver 14 }
  35. components/hab/src/lib.rs pub const VERSION: &'static str = include_str!(concat!(env!("OUT_DIR"), "/VERSION"));'

  36. Crate dependencies

  37. components/sup/Cargo.toml [dependencies] ansi_term = "*" bitflags = "*" byteorder =

    "*" clap = { version = "*", features = [ "suggestions", "color", "unstable" ] } env_logger = "*" features = "*" glob = "*" habitat_butterfly = { path = "../butterfly" } habitat_common = { path = "../common" } ...
  38. Dependency strategy: aggressive

  39. None
  40. None
  41. Cross platform codebase

  42. Cross platform tips Centralize conditional code into a few places

    Keep public API calls consistent, if possible Read the Rust codebase, it has great examples!
  43. components/core/src/os/filesyst em/mod.rs 01 #[cfg(windows)] 02 mod windows; 03 04 #[cfg(windows)]

    05 pub use self::windows::{chown, chmod, symlink}; 06 07 #[cfg(not(windows))] 08 mod linux; 09 10 #[cfg(not(windows))] 11 pub use self::linux::{chown, chmod, symlink};
  44. components/core/src/os/filesyst em/linux.rs 01 use libc::{self, c_int, c_char, mode_t}; 02 03

    pub use std::os::unix::fs::symlink; 04 use std::ffi::CString; 05 06 pub fn chown(path: &str, uid: u32, gid: u32) -> Result<c_int> { 07 // ... 08 } 09 10 pub fn chmod(path: &str, mode: u32) -> Result<c_int> { 11 // ... 12 }
  45. components/core/src/os/filesyst em/windows.rs 01 // ... 02 03 pub fn chown(path:

    &str, uid: String, gid: String) 04 -> Result<c_int> { 05 path_exists(path) 06 } 07 08 pub fn chmod(path: &str, mode: u32)-> Result<c_int> { 09 path_exists(path) 10 } 11 12 pub fn symlink<P: AsRef<Path>, Q: AsRef<Path>>(src: P, dst: Q) 13 -> io::Result<()> { 14 unimplemented!(); 15 }
  46. Onboarding team members

  47. None
  48. Onboarding tips Give new members time and space to learn,

    experiment, ask questions Pair on features Talk about architecture and code's "point of view" Talk about .unwrap() and .expect() Be patient--it gets really good if you put the time in!
  49. Rust pain points

  50. Slow compile times

  51. Hint: Cargo workspaces

  52. Cross platform code paths

  53. Hint: Continuous Integration

  54. Rust is too fast?

  55. Hint: Think about synchronization points

  56. Wrapping up

  57. Is Rust ready to be used in production? Yes, you

    bet.
  58. http://slack.habitat.sh/

  59. Thank you!