Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Workflows in the OCaml Platform

Workflows in the OCaml Platform

Describe progress towards codifying concrete supported workflows in the forthcoming OCaml Platform

Anil Madhavapeddy

August 22, 2019
Tweet

More Decks by Anil Madhavapeddy

Other Decks in Programming

Transcript

  1. Workflows in the
    OCaml Platform
    With many contributions from the OCaml community contributors and
    particularly Jeremie Dimino, Louis Gesbert, Raja Boujbel, Thomas
    Gazagnaire, Nathan Rebours for the content here

    (full list on slide 3)
    OCaml Workshop @ ICFP
    August 2019
    Anil Madhavapeddy (speaker),

    Gemma Gordon, David Allsopp

    University of Cambridge Computer Laboratory

    View Slide

  2. The OCaml Platform
    opam packaging LTS (2.0.5) LGPLv2
    dune building LTS (1.11.1) MIT DCO
    merlin editing stable (3.3.2) MIT
    utop exploring stable (2.4.1) BSD3
    ocp-indent editing stable (1.7.0) LGPLv2
    mdx testing stable (1.4.0) ISC
    bun testing devel (0.3.3) MIT
    odoc documenting devel (1.4.1) ISC
    dune-release packaging devel (1.3.2) ISC
    ocamlformat editing devel (0.11.0) MIT CLA
    in 2019

    View Slide

  3. Platform Contributors: 250+
    Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad,
    Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below,
    Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi
    Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre
    Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo
    Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien
    Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean-
    Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas
    Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu,
    Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard
    Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent
    Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek
    Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi,
    Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim
    McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le
    Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan,
    Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou,
    Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo,
    Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan
    Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg,
    Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach
    Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci,
    Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario
    Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets,
    Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John
    Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w,
    asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn,
    Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg

    View Slide

  4. The OCaml Platform makes
    users productive with the
    OCaml language
    But who are these users?
    This talk will focus on the use-cases
    we are building the OCaml tooling for

    View Slide

  5. OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    sourceforge
    Debian RedHat OpenBSD
    ocamlforge inria
    Big
    OCaml
    Project
    in 2009

    View Slide

  6. OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    sourceforge
    Debian RedHat OpenBSD
    ocamlforge inria
    Big
    OCaml
    Project
    Primary
    mechanism for
    users to get
    software
    Packaged
    with apt, rpm,
    ports, etc
    No centralised
    distribution or version
    control
    in 2009

    View Slide

  7. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    sourceforge
    Debian RedHat OpenBSD
    ocamlforge inria
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    xen
    in 2012
    opam

    View Slide

  8. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    sourceforge
    Debian RedHat OpenBSD
    ocamlforge inria
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    xen
    Centralised
    git repository of
    community code
    Chunk
    projects into
    shareable
    libraries
    in 2012
    opam

    View Slide

  9. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    in 2014
    opam

    View Slide

  10. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    in 2014
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Almost
    everyone shifts
    to GitHub for
    hosting
    Compiler
    starts moving
    packages into
    opam
    opam

    View Slide

  11. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    in 2015
    opam

    View Slide

  12. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    Lots more
    packaging work
    for OS distros
    opam used
    by most OCaml
    developers
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    in 2015
    opam
    opam CI
    tests many
    distributions on
    every PR

    View Slide

  13. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    opam
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    Reason

    ML
    in 2017
    Package Managers
    npm
    yarn
    esy

    View Slide

  14. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    opam
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    Reason

    ML
    in 2017
    Package Managers
    npm
    yarn
    esy
    Now
    packaging
    gets complex!
    Different
    workflows
    Poor opam
    is being
    overloaded

    View Slide

  15. A preponderance of
    wrapping

    View Slide

  16. make
    ocaml
    dpkg
    apt
    manual process
    of wrapping OCaml
    packages
    fixed
    versions of
    OCaml and
    libraries
    OS distro
    packaging
    done by distro
    maintainers

    View Slide

  17. make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    OS distro Dev Workflow

    View Slide

  18. make
    ocaml
    dpkg
    apt opam
    ocaml
    source
    oriented
    developer
    workflow
    opam-repo
    ocamlbuild
    multiple
    versions of
    OCaml and
    packages
    wrapping
    “make” or
    “ocamlbuild”
    OS distro Dev Workflow
    distro maintainers take
    a snapshot of opam-repo
    regularly

    View Slide

  19. make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    opam
    ocamlbuild
    opam-repo
    make
    esy
    npm
    OS distro Dev Workflow ReasonML
    ocaml

    View Slide

  20. OS distro Dev Workflow ReasonML
    make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    “distros cant keep up
    with package volume!”
    - Debian maintainer
    “pinning 100 packages
    takes a long time!”
    - MirageOS developer
    opam
    ocamlbuild
    opam-repo
    make
    esy
    npm
    ocaml
    “opam needs lock files to
    match the npm workflow!”
    - ReasonML maintainer
    “everyone is complaining
    about different things!”
    - Anil Madhavapeddy

    View Slide

  21. opam
    lib
    merlin
    lib
    debian redhat openbsd
    alpine arch macos
    npm brew
    esy
    lib
    ounit
    ppx
    mdx
    yarn
    cygwin
    wsl
    odoc
    format
    ocamlfind
    ocamlbuild
    SLOW
    ocaml

    View Slide

  22. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve

    View Slide

  23. The OCaml Platform
    Challenge #1:

    incrementally update tools without breaking any workflow
    Challenge #2:

    remove endless layers of tools wrapping tools

    Our answer: dune and opam

    - composable, declarative builds: specialised to OCaml
    - reduce choice: deprecate ocamlbuild, omake, make
    - far higher performance and reproducibility in return.
    in 2017
    in 2019

    View Slide

  24. Rearranging
    responsibilities
    opam the tool has been overloaded!
    • We have two distinct styles of workflows:
    • libraries: collections of OCaml modules
    • projects: executables, data
    • shift build logic into new tool: dune
    • packaging is what opam continues to do

    View Slide

  25. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo

    View Slide

  26. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0

    View Slide

  27. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.1.0
    install foo.1.0
    make bar.0.3
    install bar.0.3
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0

    View Slide

  28. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.1.0
    install foo.1.0
    make bar.0.3
    install bar.0.3
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt

    View Slide

  29. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.2.0
    install foo.2.0
    make bar.1.0
    install bar.1.0
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt

    View Slide

  30. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam update foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.2.0
    install foo.2.0
    make bar.1.0
    install bar.1.0
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt

    View Slide

  31. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0

    View Slide

  32. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo

    View Slide

  33. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo
    Packaging

    View Slide

  34. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo
    Packaging
    Build

    View Slide

  35. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve

    View Slide

  36. Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    opam install
    version
    constraints
    multiple valid
    choices of
    versions
    Role: Library Author

    View Slide

  37. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    git clone
    dune install
    Role: End User

    View Slide

  38. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    binary is the
    only public output
    of projects
    git clone
    dune install
    every library
    needed is in
    source tree
    Role: End User
    dubbed a
    “duniverse”

    View Slide

  39. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    Role: Project Maintainer

    View Slide

  40. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    duniverse init
    uses opam solver
    to pull libraries into
    project
    library upgrade is
    per project
    Role: Project Maintainer

    View Slide

  41. Projects
    unison.1.2.0
    tezos.2.3.0
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    Role: Distribution Maintainer

    View Slide

  42. Projects
    unison.1.2.0
    tezos.2.3.0
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    debian can solve
    for a coinstallable
    library / project set
    dune can generate
    deb package
    metadata
    Role: Distribution Maintainer

    View Slide

  43. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    esy / npm can
    vendor multiple
    versions of libraries
    dune artefact
    cache does sharing
    Role: Web developer

    View Slide

  44. merlin
    ocamlformat
    mdx ppx
    ounit
    dune
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/

    View Slide

  45. opam
    merlin
    ocamlformat
    mdx ppx
    ounit
    dune
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/

    View Slide

  46. opam
    merlin
    debian
    redhat
    *bsd
    alpine arch
    macos
    ocamlformat
    mdx ppx
    ounit
    generate
    upstreamable
    OS packages
    dune
    cygwin
    wsl
    ubuntu
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/

    View Slide

  47. opam
    merlin
    debian
    redhat
    *bsd
    alpine arch
    macos
    npm esy
    ocamlformat
    mdx ppx
    ounit
    generate
    upstreamable
    OS packages
    dune
    cygwin
    wsl
    generate package
    manager
    metadata
    ubuntu
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/
    yarn

    View Slide

  48. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve

    View Slide

  49. Next Steps: Library Author
    • Continue to use opam as you do today!
    • 1.2 -> 2.0 switch - everyone has switched now. Regular
    point releases of 2.0.x to cover portability, workflow fixes,
    sandboxing improvements.

    • New maintainer Raja Boujbel has cut recent releases,
    along with Louis Gesbert and David Allsopp.

    • Plans for opam 2.1 and 2.2 are: a new solver based on
    Z3, Windows support, integrated depext.

    View Slide

  50. Next Steps: Library Author
    0
    2000
    4000
    6000
    8000
    10000
    12000
    01/01/12 01/01/13 01/01/14 01/01/15 01/01/16 01/01/17 01/01/18 01/01/19 01/01/20
    %s
    Time
    unique opam packages

    View Slide

  51. Next Steps: Library Author
    • Existing CI system is popular and useful, but
    cumbersome to use
    • https://ci.ocaml.org/
    • Working on next generation CI to make life easier:
    • https://github.com/ocaml-ci/

    • https://ci.ocamllabs.io

    • Based on a DSL similar to dune rules, so can express
    complex workflows.

    • Term language that can work with CLI and web

    • Automate bug triage — revdep breakage

    • Easy to extend with Discourse, Slack, etc

    View Slide

  52. Next Steps: End User
    • Projects using duniverse now:
    • https://github.com/realworldocaml/book
    • https://github.com/avsm/duniverse
    • https://github.com/avsm/platform
    • Dune has been one of the fastest adopted
    Platform efforts since opam.
    • git clone; dune build

    View Slide

  53. Next Steps: End User
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    building
    "duniverse"
    monorepo for
    Platform project
    mdx

    View Slide

  54. Next Steps: End User
    • Coming soon in dune 2.0:

    dune artefact cache
    • https://github.com/ocaml/dune/pull/2493
    • Reliable per-rule caching of binary artefacts
    • Works locally first, eventually networking
    • Will eventually support reproducible builds
    throughout our stack, and therefore safe
    networked binary caches of OCaml code.

    View Slide

  55. Next Steps: Project Maintainer
    • duniverse tool to bridge opam and dune
    https://github.com/avsm/duniverse
    • currently supports:
    • duniverse init: turn opam packages into git refs
    • duniverse pull: fetch git refs to local tree
    • coming soon:
    • duniverse update: merge new opam packages
    • duniverse install: add new opam packages

    View Slide

  56. Next Steps: Project Maintainer
    • Do you have to use dune to get these benefits?

    View Slide

  57. Next Steps: Project Maintainer
    Yes

    View Slide

  58. Next Steps: Project Maintainer
    • Do you have to use dune to get these benefits?
    • Yes, but we maintain a transition remote:
    • https://github.com/dune-universe/opam-overlays
    • duniverse prototype tool automates inserting
    overrides.
    • We will provide a formal specification of dune
    files, so other implementations can be built.

    View Slide

  59. Next Steps: Project Maintainer
    commit 632bc2eed00704bbbcdb5531d709c8d3f0cb0939
    Author: Thomas Gazagnaire
    Date: Wed Mar 28 15:49:16 2012 +0200
    Initial commit
    commit db943c24fe3ed5cec4dd432b1a01c66602d5f2f3
    Author: Xavier Leroy
    Date: Sun Jul 24 19:04:21 2016 +0200
    Add depexts for CentOS et al
    ~4 years
    opam-repo

    View Slide

  60. Next Steps: Project Maintainer
    commit 632bc2eed00704bbbcdb5531d709c8d3f0cb0939
    Author: Thomas Gazagnaire
    Date: Wed Mar 28 15:49:16 2012 +0200
    Initial commit
    commit db943c24fe3ed5cec4dd432b1a01c66602d5f2f3
    Author: Xavier Leroy
    Date: Sun Jul 24 19:04:21 2016 +0200
    Add depexts for CentOS et al
    ~4 years
    commit 93b5d9bdb9c25d1ea86a96a9c9fce93ecebd5c6d
    Author: Jeremie Dimino
    Date: Sun Oct 30 19:07:53 2016 +0000
    initial import
    opam-repo
    dune

    View Slide

  61. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage, why3, frama-c)
    • ocaml 4.08.x has early dune files

    View Slide

  62. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage, why3, frama-c)
    • ocaml 4.08.x has early dune files
    (lang dune 1.10)
    ;; every metadata file is versioned

    View Slide

  63. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage)
    • ocaml 4.08.x has early dune files
    CHANGES in one year:
    1.2.0: polling mode for build, private modules
    1.3.0: robust merlin file generation, colors on Windows!
    1.4.0: automatic code formatter targets, menhir rules
    1.5.0: @check target for rapid type checking, cross compilation for ocamlfind
    1.6.0: more flexible directory layouts for embedding in other projects
    1.7.0: improved optimisation for cmx, install metadata, formally specified cycle detection
    1.8.0: code promotion improvements, better c++ support, merlin on Windows
    1.9.0: library variants and default implementations, coq support
    1.10.0: generate opam metadata, future syntax shims, compile-time selection of os/arch
    1.11.0: dune init, cinaps support, build version embedding, inline test improvements

    View Slide

  64. Next Steps: Distribution Maintainer
    • Our Linux distribution friends need more
    assistance!
    • Historically they have had good tooling to
    bridge, but now need better opam/dune
    integration.
    • Challenging new project in 2020 is Deb/RPM
    generation from duniverses.

    View Slide

  65. Next Steps: Distribution Maintainer
    • Vision: multiple coinstallable and buildable
    trees that represent all of Debian’s or
    CentOS’s OCaml code in one git tree.
    • This is incredibly essential for the future of
    OCaml development.
    • A single dune build can test tens of millions
    of lines of OCaml code with multicore,
    flambda, algebraic effects and other new
    features in OCaml.

    View Slide

  66. Next Steps: web/mobile/unikernel
    developer
    • duniverses make cross compilation to exotic
    targets extremely viable.
    • see talk on MirageOS 4 later today!
    • wrapping just a build system like dune frees
    esy / npm to do their own packaging
    • huge ecosystem potential of JavaScript,
    wasm, iOS/Android, unikernel, unlocked by
    OCaml

    View Slide

  67. The OCaml Platform
    opam LTS (2.0.5) LGPLv2
    dune LTS (1.11.1) MIT DCO
    merlin stable (3.3.2) MIT
    utop stable (2.4.1) BSD3
    ocp-indent stable (1.7.0) LGPLv2
    mdx devel (1.4.0) ISC
    bun devel (0.3.3) MIT
    odoc devel (1.4.1) ISC
    dune-release devel (1.3.2) ISC
    ocamlformat devel (0.11.0) MIT CLA
    in 2019
    discuss.ocaml.org

    View Slide

  68. The OCaml Platform
    in 2019…
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    mdx
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    Closer
    integration
    between tools
    dune-release
    building

    View Slide

  69. Contact Lead Maintainers
    opam LTS (2.0.5) Raja Boujbel, Louis Gesbert
    dune LTS (1.11.1) Jeremie Dimino, Rudi Grinberg
    merlin stable (3.3.2) Fred Bour, Thomas Refis
    utop stable (2.4.1) Jeremie Dimino
    ocp-indent stable (1.7.0) Louis Gesbert
    mdx devel (1.4.0) Thomas Gazagnaire
    bun devel (0.3.3) Mindy Preston
    odoc devel (1.4.1) Anton Bachin, Jon Ludlam
    dune-release devel (1.3.2) Thomas Gazagnaire, Nathan Rebours
    ocamlformat devel (0.11.0) Josh Berdine, Guillaume Petiot
    in 2019
    discuss.ocaml.org

    View Slide

  70. The OCaml Platform
    in 2019…
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    building
    "duniverse"
    monorepo for
    Platform
    project
    mdx

    View Slide

  71. The OCaml Platform
    opam LTS (2.0.5) LGPLv2
    dune LTS (1.11.1) MIT DCO
    merlin stable (3.3.2) MIT
    utop stable (2.4.1) BSD3
    ocp-indent stable (1.7.0) LGPLv2
    mdx devel (1.4.0) ISC
    bun devel (0.3.3) MIT
    odoc devel (1.4.1) ISC
    dune-release devel (1.3.2) ISC
    ocamlformat devel (0.11.0) MIT CLA

    View Slide