Nix for Scala folks

Quick intro to Nix and optimal usage with the Scala ecosystem in 2022.

Transcript

1. Nix for Scala folks Jakub Kozłowski, Disney Streaming | 11.05.2022 | Warsaw
2. Agenda Two words about Nix Nix in the Scala world Nix+Scala in the future
3. Agenda Two words about Nix Nix in the Scala world Nix+Scala in the future
4. What is Nix?
5. What is Nix? • Package manager
6. What is Nix? • Package manager • Build system
7. What is Nix? • Package manager • Build system • Language
8. What is Nix? • Package manager • Build system • Language • Linux distro (NixOS)
9. Nix is a scalable tool for building and managing packages in a reproducible fashion. tl;dr
10. The main repository with Nix package de f initions github.com/nixos/nixpkgs Nixpkgs
11. Agenda Two words about Nix Nix in the Scala world Nix+Scala in the future
12. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt)
13. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) What needs to be compiled
14. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) git, node, awscli, docker-client, ...
15. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) JAVA_OPTS, SBT_OPTS, ...
16. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) build.sbt, project/*
17. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) *.graphql, *.proto, *.smithy
18. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) runs sbt, compiler, tests...
19. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) scala-compiler-2.13.6.jar
20. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) cats-e ff ect-3.3.0.jar fs2-core-3.2.0.jar
21. But f irst... typical sbt build inputs (let's be honest, everyone uses sbt) bin/sbt, sbt-1.6.2.jar
22. Environment setup
23. Environment setup with Nix let pkgs = import (builtins.fetchTarball { url = "https: / / github.com/NixOS/nixpkgs/archive/20eabe33864104ad9928450e5927ab7835f8f9e8.tar.gz"; sha256 = "0l00xjjd436ddq1llfvpcipz06b6s0zxqrmfm9mgj7c0ig4y4c0r"; }) { }; in pkgs.mkShell { packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }
24. Environment setup with Nix pkgs.mkShell { packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }
25. Environment setup with Nix
26. Why bother? • Doesn't pollute your environment 🌳
27. Why bother? • Doesn't pollute your environment 🌳 • Doesn't con f lict with globally installed programs
28. Why bother? • Doesn't pollute your environment 🌳 • Doesn't con f lict with globally installed programs • Project-speci f ic version of the JVM/terraform/awscli/node/...
29. Why bother? • Doesn't pollute your environment 🌳 • Doesn't con f lict with globally installed programs • Project-speci f ic version of the JVM/terraform/awscli/node/... • Share tools with the team (Git)
30. Why bother? • Doesn't pollute your environment 🌳 • Doesn't con f lict with globally installed programs • Project-speci f ic version of the JVM/terraform/awscli/node/... • Share tools with the team (Git) • Same setup locally and in Continuous Integration
31. Environment variables in Nix shell pkgs.mkShell { packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; JAVA_OPTS = "-Xmx4G"; }
32. Temporary shells
33. Bonus: nix-direnv
34. Bonus: f lakes { inputs.nixpkgs.url = "github:nixos/nixpkgs"; inputs.flake-utils.url = "github:numtide/flake-utils"; outputs = { self, nixpkgs, flake-utils, . .. }: }
35. Bonus: f lakes flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; in );
36. Bonus: f lakes { devShells.default = pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }; }
37. Bonus: f lakes { inputs.nixpkgs.url = "github:nixos/nixpkgs"; inputs.flake-utils.url = "github:numtide/flake-utils"; outputs = { self, nixpkgs, flake-utils, . .. }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; in { devShells.default = pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }; } ); }
38. Bonus: f lakes • Uses lock f ile to specify hash of Nixpkgs and other repositories { inputs.nixpkgs.url = "github:nixos/nixpkgs"; inputs.flake-utils.url = "github:numtide/flake-utils"; outputs = { self, nixpkgs, flake-utils, . .. }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; in { devShells.default = pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }; } ); }
39. Bonus: f lakes • Uses lock f ile to specify hash of Nixpkgs and other repositories • Less boilerplate for pinning { inputs.nixpkgs.url = "github:nixos/nixpkgs"; inputs.flake-utils.url = "github:numtide/flake-utils"; outputs = { self, nixpkgs, flake-utils, . .. }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; in { devShells.default = pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }; } ); }
40. Bonus: f lakes • Uses lock f ile to specify hash of Nixpkgs and other repositories • Less boilerplate for pinning • Still experimental, e ff orts to further simplify { inputs.nixpkgs.url = "github:nixos/nixpkgs"; inputs.flake-utils.url = "github:numtide/flake-utils"; outputs = { self, nixpkgs, flake-utils, . .. }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; in { devShells.default = pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; }; } ); }
41. See also: devshell • TOML-based • Nix language not needed for 80% use cases • Experimental https://github.com/numtide/devshell [devshell] packages = [ "go" ]
42. Installing packages
43. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER
44. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER • Atomic installation, rollbacks
45. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER • Atomic installation, rollbacks • No transitive con f licts
46. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER • Atomic installation, rollbacks • No transitive con f licts • User-scoped
47. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER • Atomic installation, rollbacks • No transitive con f licts • User-scoped • For a declarative approach, see home-manager
48. nix pro f ile • Drop-in replacement for $PACKAGE_MANAGER • Atomic installation, rollbacks • No transitive con f licts • User-scoped • For a declarative approach, see home-manager • Also experimental, see nix-env
49. Recommendations
50. Recommendations Before things get out of hand
51. Recommendations • Use nix shells in your projects!
52. Recommendations • Use nix shells in your projects! • Use one-time shells to try stu ff out
53. Recommendations • Use nix shells in your projects! • Use one-time shells to try stu ff out • Flakes + shells + nix-direnv = 🥰
54. Recommendations • Use nix shells in your projects! • Use one-time shells to try stu ff out • Flakes + shells + nix-direnv = 🥰 • Use nix as a normal package manager
55. Recommendations • Use nix shells in your projects! • Use one-time shells to try stu ff out • Flakes + shells + nix-direnv = 🥰 • Use nix as a normal package manager • Use shells in CI
57. More use cases for Nix • System/user program management • Declarative system con f iguration • CI pipelines • Deployment • Building small docker images • ??? • ???
58. Build system
59. Nix as a build system def makePackage( dependencies: List[Package], src: Path, buildScript: String, ): Package = ??? (simpli f ied)
60. Nix as a build system def makePackage( dependencies: List[Package], src: Path, buildScript: String, ): Package = ??? (simpli f ied) • No access to internet*, other f iles on disk • Build script has to output f iles to a speci f ic location
61. Nix as a build system def makePackage( dependencies: List[Package], src: Path, buildScript: String, ): Package = ??? (simpli f ied) • No access to internet*, other f iles on disk • Build script has to output f iles to a speci f ic location * - Internet can be used if you know the output's hash beforehand
62. Typical sbt build inputs (again)
63. State of the art
64. State of the art • Single dependency hash
65. State of the art • Single dependency hash • Waste of space (duplicated deps)
66. State of the art • Single dependency hash • Waste of space (duplicated deps) • Slow, non-incremental
67. Agenda Two words about Nix Nix in the Scala world Nix+Scala in the future
68. Inspiration
69. Inspiration: Rust Nix-friendly lock f ile based on Cargo.lock
70. Inspiration: Rust Dependencies are built in isolated, parallel Nix builds
71. Hope
72. Nix+Scala in a perfect world
73. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line
74. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches
75. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches • Saving space with dependency reuse
76. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches • Saving space with dependency reuse • Splitting sbt build into modules?
77. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches • Saving space with dependency reuse • Splitting sbt build into modules? • Direct integration with scalac/bloop?
78. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches • Saving space with dependency reuse • Splitting sbt build into modules? • Direct integration with scalac/bloop? • Replacing sbt's dependency resolution?
79. Nix+Scala in a perfect world • Lock f ile generated from sbt with a single line • Fast setup, using existing caches • Saving space with dependency reuse • Splitting sbt build into modules? • Direct integration with scalac/bloop? • Replacing sbt's dependency resolution? • New build tool?
80. Nix+Scala work today
81. Nix+Scala work today • sbt-derivation (not actively developed)
82. Nix+Scala work today • sbt-derivation (not actively developed) • Open ticket in sbt-derivation to support lock f iles
83. Nix+Scala work today • sbt-derivation (not actively developed) • Open ticket in sbt-derivation to support lock f iles • Possibly needed: new feature in sbt
84. How to get started? sh <(curl -L https: // nixos.org/nix/install) -- daemon • https://nixos.org/guides/nix-pills + https://nixos.org/manual/nix/stable/ • https://nix.dev • I'm writing a book! Follow https://leanpub.com/nix-book The only time you'll ever have to sh from curl :)
85. Thank you! 📰 blog.kubukoz.com 🐦 @kubukoz Slides: speakerdeck.com/kubukoz YouTube: yt.kubukoz.com