Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Nix for Scala folks

Nix for Scala folks

Quick intro to Nix and optimal usage with the Scala ecosystem in 2022.

Jakub Kozłowski

May 11, 2022
Tweet

More Decks by Jakub Kozłowski

Other Decks in Programming

Transcript

  1. Nix for Scala folks
    Jakub Kozłowski, Disney Streaming | 11.05.2022 | Warsaw

    View Slide

  2. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future

    View Slide

  3. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future

    View Slide

  4. What is Nix?

    View Slide

  5. What is Nix?
    • Package manager

    View Slide

  6. What is Nix?
    • Package manager
    • Build system

    View Slide

  7. What is Nix?
    • Package manager
    • Build system
    • Language

    View Slide

  8. What is Nix?
    • Package manager
    • Build system
    • Language
    • Linux distro (NixOS)

    View Slide

  9. Nix is a scalable tool for building
    and managing packages in a
    reproducible fashion.
    tl;dr

    View Slide

  10. The main repository with Nix package de
    f
    initions


    github.com/nixos/nixpkgs
    Nixpkgs

    View Slide

  11. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future

    View Slide

  12. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)

    View Slide

  13. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    What needs to be
    compiled

    View Slide

  14. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    git, node, awscli,
    docker-client, ...

    View Slide

  15. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    JAVA_OPTS,
    SBT_OPTS, ...

    View Slide

  16. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    build.sbt, project/*

    View Slide

  17. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    *.graphql,
    *.proto, *.smithy

    View Slide

  18. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    runs sbt,
    compiler, tests...

    View Slide

  19. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    scala-compiler-2.13.6.jar

    View Slide

  20. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    cats-e
    ff
    ect-3.3.0.jar


    fs2-core-3.2.0.jar

    View Slide

  21. But
    f
    irst... typical sbt build inputs
    (let's be honest, everyone uses sbt)
    bin/sbt, sbt-1.6.2.jar

    View Slide

  22. Environment setup

    View Slide

  23. Environment setup with Nix
    let pkgs = import


    (builtins.fetchTarball {


    url = "https:
    /
    /
    github.com/NixOS/nixpkgs/archive/20eabe33864104ad9928450e5927ab7835f8f9e8.tar.gz";


    sha256 = "0l00xjjd436ddq1llfvpcipz06b6s0zxqrmfm9mgj7c0ig4y4c0r";


    })


    { }; in


    pkgs.mkShell {


    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];


    }

    View Slide

  24. Environment setup with Nix
    pkgs.mkShell {


    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];


    }

    View Slide

  25. Environment setup with Nix

    View Slide

  26. Why bother?
    • Doesn't pollute your environment 🌳

    View Slide

  27. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs

    View Slide

  28. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...

    View Slide

  29. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...
    • Share tools with the team (Git)

    View Slide

  30. Why bother?
    • Doesn't pollute your environment 🌳
    • Doesn't con
    f
    lict with globally installed programs
    • Project-speci
    f
    ic version of the JVM/terraform/awscli/node/...
    • Share tools with the team (Git)
    • Same setup locally and in Continuous Integration

    View Slide

  31. Environment variables in Nix shell
    pkgs.mkShell {


    packages = [ pkgs.sbt pkgs.coursier pkgs.nodejs ];


    JAVA_OPTS = "-Xmx4G";


    }

    View Slide

  32. Temporary shells

    View Slide

  33. Bonus: nix-direnv

    View Slide

  34. Bonus:
    f
    lakes
    {


    inputs.nixpkgs.url = "github:nixos/nixpkgs";


    inputs.flake-utils.url = "github:numtide/flake-utils";


    outputs = { self, nixpkgs, flake-utils,
    . ..
    }:
    }

    View Slide

  35. Bonus:
    f
    lakes
    flake-utils.lib.eachDefaultSystem (system:


    let


    pkgs = import nixpkgs { inherit system; };


    in


    );


    View Slide

  36. Bonus:
    f
    lakes
    {


    devShells.default =


    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };


    }


    View Slide

  37. Bonus:
    f
    lakes
    {


    inputs.nixpkgs.url = "github:nixos/nixpkgs";


    inputs.flake-utils.url = "github:numtide/flake-utils";


    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:


    let


    pkgs = import nixpkgs { inherit system; };


    in


    {


    devShells.default =


    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };


    }


    );


    }

    View Slide

  38. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    {


    inputs.nixpkgs.url = "github:nixos/nixpkgs";


    inputs.flake-utils.url = "github:numtide/flake-utils";


    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:


    let


    pkgs = import nixpkgs { inherit system; };


    in


    {


    devShells.default =


    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };


    }


    );


    }

    View Slide

  39. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    • Less boilerplate for pinning
    {


    inputs.nixpkgs.url = "github:nixos/nixpkgs";


    inputs.flake-utils.url = "github:numtide/flake-utils";


    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:


    let


    pkgs = import nixpkgs { inherit system; };


    in


    {


    devShells.default =


    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };


    }


    );


    }

    View Slide

  40. Bonus:
    f
    lakes
    • Uses lock
    f
    ile to specify hash of Nixpkgs and other repositories
    • Less boilerplate for pinning
    • Still experimental, e
    ff
    orts to further simplify
    {


    inputs.nixpkgs.url = "github:nixos/nixpkgs";


    inputs.flake-utils.url = "github:numtide/flake-utils";


    outputs = { self, nixpkgs, flake-utils,
    . ..
    }: flake-utils.lib.eachDefaultSystem (system:


    let


    pkgs = import nixpkgs { inherit system; };


    in


    {


    devShells.default =


    pkgs.mkShell { buildInputs = [ pkgs.sbt pkgs.coursier pkgs.nodejs ]; };


    }


    );


    }

    View Slide

  41. See also: devshell
    • TOML-based


    • Nix language not needed for 80%
    use cases


    • Experimental


    https://github.com/numtide/devshell
    [devshell]


    packages = [


    "go"


    ]

    View Slide

  42. Installing packages

    View Slide

  43. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER

    View Slide

  44. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks

    View Slide

  45. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts

    View Slide

  46. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped

    View Slide

  47. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped
    • For a declarative approach, see
    home-manager

    View Slide

  48. nix pro
    f
    ile
    • Drop-in replacement for
    $PACKAGE_MANAGER
    • Atomic installation, rollbacks
    • No transitive con
    f
    licts
    • User-scoped
    • For a declarative approach, see
    home-manager
    • Also experimental, see nix-env

    View Slide

  49. Recommendations

    View Slide

  50. Recommendations
    Before things get out of hand

    View Slide

  51. Recommendations
    • Use nix shells in your projects!

    View Slide

  52. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out

    View Slide

  53. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰

    View Slide

  54. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰
    • Use nix as a normal package manager

    View Slide

  55. Recommendations
    • Use nix shells in your projects!
    • Use one-time shells to try stu
    ff
    out
    • Flakes + shells + nix-direnv = 🥰
    • Use nix as a normal package manager
    • Use shells in CI

    View Slide

  56. View Slide

  57. More use cases for Nix
    • System/user program management


    • Declarative system con
    f
    iguration


    • CI pipelines


    • Deployment


    • Building small docker images


    • ???


    • ???

    View Slide

  58. Build system

    View Slide

  59. Nix as a build system
    def makePackage(


    dependencies: List[Package],


    src: Path,


    buildScript: String,


    ): Package =
    ???
    (simpli
    f
    ied)

    View Slide

  60. Nix as a build system
    def makePackage(


    dependencies: List[Package],


    src: Path,


    buildScript: String,


    ): Package =
    ???
    (simpli
    f
    ied)
    • No access to internet*, other
    f
    iles on
    disk


    • Build script has to output
    f
    iles to a
    speci
    f
    ic location

    View Slide

  61. Nix as a build system
    def makePackage(


    dependencies: List[Package],


    src: Path,


    buildScript: String,


    ): Package =
    ???
    (simpli
    f
    ied)
    • No access to internet*, other
    f
    iles on
    disk


    • Build script has to output
    f
    iles to a
    speci
    f
    ic location
    * - Internet can be used if you know the output's hash beforehand

    View Slide

  62. Typical sbt build inputs
    (again)

    View Slide

  63. State of the art

    View Slide

  64. State of the art
    • Single dependency hash

    View Slide

  65. State of the art
    • Single dependency hash
    • Waste of space (duplicated deps)

    View Slide

  66. State of the art
    • Single dependency hash
    • Waste of space (duplicated deps)
    • Slow, non-incremental

    View Slide

  67. Agenda
    Two words about Nix Nix in the Scala world Nix+Scala in the future

    View Slide

  68. Inspiration

    View Slide

  69. Inspiration: Rust
    Nix-friendly lock
    f
    ile based on Cargo.lock

    View Slide

  70. Inspiration: Rust
    Dependencies are built in isolated, parallel Nix builds

    View Slide

  71. Hope

    View Slide

  72. Nix+Scala in a perfect world

    View Slide

  73. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line

    View Slide

  74. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches

    View Slide

  75. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse

    View Slide

  76. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?

    View Slide

  77. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?

    View Slide

  78. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?
    • Replacing sbt's dependency resolution?

    View Slide

  79. Nix+Scala in a perfect world
    • Lock
    f
    ile generated from sbt with a single line
    • Fast setup, using existing caches
    • Saving space with dependency reuse
    • Splitting sbt build into modules?
    • Direct integration with scalac/bloop?
    • Replacing sbt's dependency resolution?
    • New build tool?

    View Slide

  80. Nix+Scala work today

    View Slide

  81. Nix+Scala work today
    • sbt-derivation (not actively developed)

    View Slide

  82. Nix+Scala work today
    • sbt-derivation (not actively developed)
    • Open ticket in sbt-derivation to support lock
    f
    iles

    View Slide

  83. Nix+Scala work today
    • sbt-derivation (not actively developed)
    • Open ticket in sbt-derivation to support lock
    f
    iles
    • Possibly needed: new feature in sbt

    View Slide

  84. How to get started?
    sh <(curl -L https:
    //
    nixos.org/nix/install)
    --
    daemon


    • https://nixos.org/guides/nix-pills + https://nixos.org/manual/nix/stable/


    • https://nix.dev


    • I'm writing a book! Follow https://leanpub.com/nix-book
    The only time you'll ever have to sh from curl :)

    View Slide

  85. Thank you!
    📰 blog.kubukoz.com


    🐦 @kubukoz


    Slides: speakerdeck.com/kubukoz


    YouTube: yt.kubukoz.com

    View Slide