Nix for Scala folks

Quick intro to Nix and optimal usage with the Scala ecosystem in 2022.

Transcript

1. Nix for Scala folks
   Jakub Kozłowski, Disney Streaming | 11.05.2022 | Warsaw
   

   View Slide

2. Agenda
   Two words about Nix Nix in the Scala world Nix+Scala in the future
   

   View Slide

3. Agenda
   Two words about Nix Nix in the Scala world Nix+Scala in the future
   

   View Slide

4. What is Nix?
   

   View Slide

5. What is Nix?
   • Package manager
   

   View Slide

6. What is Nix?
   • Package manager
   • Build system
   

   View Slide

7. What is Nix?
   • Package manager
   • Build system
   • Language
   

   View Slide

8. What is Nix?
   • Package manager
   • Build system
   • Language
   • Linux distro (NixOS)
   

   View Slide

9. Nix is a scalable tool for building
   and managing packages in a
   reproducible fashion.
   tl;dr
   

   View Slide

10. The main repository with Nix package de
    f
    initions
    
    
    github.com/nixos/nixpkgs
    Nixpkgs
    

    View Slide

11. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future
    

    View Slide

12. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    

    View Slide

13. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    What needs to be
    compiled
    

    View Slide

14. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    git, node, awscli,
    docker-client, ...
    

    View Slide

15. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    JAVA_OPTS,
    SBT_OPTS, ...
    

    View Slide

16. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    build.sbt, project/*
    

    View Slide

17. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    *.graphql,
    *.proto, *.smithy
    

    View Slide

18. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    runs sbt,
    compiler, tests...
    

    View Slide

19. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    scala-compiler-2.13.6.jar
    

    View Slide

20. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    cats-e
    ff
    ect-3.3.0.jar
    
    
    fs2-core-3.2.0.jar
    

    View Slide

21. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    bin/sbt, sbt-1.6.2.jar
    

    View Slide

22. Environment setup
    

    View Slide

23. Environment setup with Nix
    let pkgs = import
    
    
    (builtins.fetchTarball {
    
    
    url = "https:
    /
    /
    github.com/NixOS/nixpkgs/archive/20eabe33864104ad9928450e5927ab7835f8f9e8.tar.gz";
    
    
    sha256 = "0l00xjjd436ddq1llfvpcipz06b6s0zxqrmfm9mgj7c0ig4y4c0r";
    
    
    })
    
    
    { }; in
    
    
    pkgs.mkShell {
    
    
    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];
    
    
    }
    

    View Slide

24. Environment setup with Nix
    pkgs.mkShell {
    
    
    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];
    
    
    }
    

    View Slide

25. Environment setup with Nix
    

    View Slide

26. Why bother?
    • Doesn't pollute your environment 🌳
    

    View Slide

27. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    

    View Slide

28. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...
    

    View Slide

29. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...
    • Share tools with the team (Git)
    

    View Slide

30. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...
    • Share tools with the team (Git)
    • Same setup locally and in Continuous Integration
    

    View Slide

31. Environment variables in Nix shell
    pkgs.mkShell {
    
    
    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];
    
    
    JAVA_OPTS = "-Xmx4G";
    
    
    }
    

    View Slide

32. Temporary shells
    

    View Slide

33. Bonus: nix-direnv
    

    View Slide

34. Bonus:
    f
    lakes
    {
    
    
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    
    
    inputs.flake-utils.url = "github:numtide/flake-utils";
    
    
    outputs = { self, nixpkgs, flake-utils,
    . ..
    }:
    }
    

    View Slide

35. Bonus:
    f
    lakes
    flake-utils.lib.eachDefaultSystem (system:
    
    
    let
    
    
    pkgs = import nixpkgs { inherit system; };
    
    
    in
    
    
    );
    
    
    

    View Slide

36. Bonus:
    f
    lakes
    {
    
    
    devShells.default =
    
    
    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };
    
    
    }
    
    
    

    View Slide

37. Bonus:
    f
    lakes
    {
    
    
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    
    
    inputs.flake-utils.url = "github:numtide/flake-utils";
    
    
    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:
    
    
    let
    
    
    pkgs = import nixpkgs { inherit system; };
    
    
    in
    
    
    {
    
    
    devShells.default =
    
    
    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };
    
    
    }
    
    
    );
    
    
    }
    

    View Slide

38. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    {
    
    
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    
    
    inputs.flake-utils.url = "github:numtide/flake-utils";
    
    
    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:
    
    
    let
    
    
    pkgs = import nixpkgs { inherit system; };
    
    
    in
    
    
    {
    
    
    devShells.default =
    
    
    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };
    
    
    }
    
    
    );
    
    
    }
    

    View Slide

39. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    • Less boilerplate for pinning
    {
    
    
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    
    
    inputs.flake-utils.url = "github:numtide/flake-utils";
    
    
    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:
    
    
    let
    
    
    pkgs = import nixpkgs { inherit system; };
    
    
    in
    
    
    {
    
    
    devShells.default =
    
    
    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };
    
    
    }
    
    
    );
    
    
    }
    

    View Slide

40. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    • Less boilerplate for pinning
    • Still experimental, e
    ff
    orts to further simplify
    {
    
    
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    
    
    inputs.flake-utils.url = "github:numtide/flake-utils";
    
    
    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:
    
    
    let
    
    
    pkgs = import nixpkgs { inherit system; };
    
    
    in
    
    
    {
    
    
    devShells.default =
    
    
    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };
    
    
    }
    
    
    );
    
    
    }
    

    View Slide

41. See also: devshell
    • TOML-based
    
    
    • Nix language not needed for 80%
    use cases
    
    
    • Experimental
    
    
    https://github.com/numtide/devshell
    [devshell]
    
    
    packages = [
    
    
    "go"
    
    
    ]
    

    View Slide

42. Installing packages
    

    View Slide

43. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    

    View Slide

44. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    

    View Slide

45. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    

    View Slide

46. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped
    

    View Slide

47. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped
    • For a declarative approach, see
    home-manager
    

    View Slide

48. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped
    • For a declarative approach, see
    home-manager
    • Also experimental, see nix-env
    

    View Slide

49. Recommendations
    

    View Slide

50. Recommendations
    Before things get out of hand
    

    View Slide

51. Recommendations
    • Use nix shells in your projects!
    

    View Slide

52. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    

    View Slide

53. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰
    

    View Slide

54. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰
    • Use nix as a normal package manager
    

    View Slide

55. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰
    • Use nix as a normal package manager
    • Use shells in CI
    

    View Slide

56. View Slide

57. More use cases for Nix
    • System/user program management
    
    
    • Declarative system con
    f
    iguration
    
    
    • CI pipelines
    
    
    • Deployment
    
    
    • Building small docker images
    
    
    • ???
    
    
    • ???
    

    View Slide

58. Build system
    

    View Slide

59. Nix as a build system
    def makePackage(
    
    
    dependencies: List[Package],
    
    
    src: Path,
    
    
    buildScript: String,
    
    
    ): Package =
    ???
    (simpli
    f
    ied)
    

    View Slide

60. Nix as a build system
    def makePackage(
    
    
    dependencies: List[Package],
    
    
    src: Path,
    
    
    buildScript: String,
    
    
    ): Package =
    ???
    (simpli
    f
    ied)
    • No access to internet*, other
    f
    iles on
    disk
    
    
    • Build script has to output
    f
    iles to a
    speci
    f
    ic location
    

    View Slide

61. Nix as a build system
    def makePackage(
    
    
    dependencies: List[Package],
    
    
    src: Path,
    
    
    buildScript: String,
    
    
    ): Package =
    ???
    (simpli
    f
    ied)
    • No access to internet*, other
    f
    iles on
    disk
    
    
    • Build script has to output
    f
    iles to a
    speci
    f
    ic location
    * - Internet can be used if you know the output's hash beforehand
    

    View Slide

62. Typical sbt build inputs
    (again)
    

    View Slide

63. State of the art
    

    View Slide

64. State of the art
    • Single dependency hash
    

    View Slide

65. State of the art
    • Single dependency hash
    • Waste of space (duplicated deps)
    

    View Slide

66. State of the art
    • Single dependency hash
    • Waste of space (duplicated deps)
    • Slow, non-incremental
    

    View Slide

67. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future
    

    View Slide

68. Inspiration
    

    View Slide

69. Inspiration: Rust
    Nix-friendly lock
    f
    ile based on Cargo.lock
    

    View Slide

70. Inspiration: Rust
    Dependencies are built in isolated, parallel Nix builds
    

    View Slide

71. Hope
    

    View Slide

72. Nix+Scala in a perfect world
    

    View Slide

73. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    

    View Slide

74. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    

    View Slide

75. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    

    View Slide

76. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    

    View Slide

77. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?
    

    View Slide

78. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?
    • Replacing sbt's dependency resolution?
    

    View Slide

79. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?
    • Replacing sbt's dependency resolution?
    • New build tool?
    

    View Slide

80. Nix+Scala work today
    

    View Slide

81. Nix+Scala work today
    • sbt-derivation (not actively developed)
    

    View Slide

82. Nix+Scala work today
    • sbt-derivation (not actively developed)
    • Open ticket in sbt-derivation to support lock
    f
    iles
    

    View Slide

83. Nix+Scala work today
    • sbt-derivation (not actively developed)
    • Open ticket in sbt-derivation to support lock
    f
    iles
    • Possibly needed: new feature in sbt
    

    View Slide

84. How to get started?
    sh <(curl -L https:
    //
    nixos.org/nix/install)
    --
    daemon
    
    
    • https://nixos.org/guides/nix-pills + https://nixos.org/manual/nix/stable/
    
    
    • https://nix.dev
    
    
    • I'm writing a book! Follow https://leanpub.com/nix-book
    The only time you'll ever have to sh from curl :)
    

    View Slide

85. Thank you!
    📰 blog.kubukoz.com
    
    
    🐦 @kubukoz
    
    
    Slides: speakerdeck.com/kubukoz
    
    
    YouTube: yt.kubukoz.com
    

    View Slide