Workflows in the OCaml Platform

Describe progress towards codifying concrete supported workflows in the forthcoming OCaml Platform

Transcript

1. Workflows in the
   OCaml Platform
   With many contributions from the OCaml community contributors and
   particularly Jeremie Dimino, Louis Gesbert, Raja Boujbel, Thomas
   Gazagnaire, Nathan Rebours for the content here

   (full list on slide 3)
   OCaml Workshop @ ICFP
   August 2019
   Anil Madhavapeddy (speaker),

   Gemma Gordon, David Allsopp

   University of Cambridge Computer Laboratory
   

   View Slide

2. The OCaml Platform
   opam packaging LTS (2.0.5) LGPLv2
   dune building LTS (1.11.1) MIT DCO
   merlin editing stable (3.3.2) MIT
   utop exploring stable (2.4.1) BSD3
   ocp-indent editing stable (1.7.0) LGPLv2
   mdx testing stable (1.4.0) ISC
   bun testing devel (0.3.3) MIT
   odoc documenting devel (1.4.1) ISC
   dune-release packaging devel (1.3.2) ISC
   ocamlformat editing devel (0.11.0) MIT CLA
   in 2019
   

   View Slide

3. Platform Contributors: 250+
   Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad,
   Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below,
   Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi
   Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre
   Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo
   Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien
   Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean-
   Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas
   Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu,
   Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard
   Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent
   Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek
   Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi,
   Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim
   McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le
   Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan,
   Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou,
   Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo,
   Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan
   Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg,
   Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach
   Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci,
   Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario
   Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets,
   Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John
   Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w,
   asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn,
   Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg
   

   View Slide

4. The OCaml Platform makes
   users productive with the
   OCaml language
   But who are these users?
   This talk will focus on the use-cases
   we are building the OCaml tooling for
   

   View Slide

5. OS Distributions
   Big
   OCaml
   Project
   OCaml
   compiler
   sourceforge
   Debian RedHat OpenBSD
   ocamlforge inria
   Big
   OCaml
   Project
   in 2009
   

   View Slide

6. OS Distributions
   Big
   OCaml
   Project
   OCaml
   compiler
   sourceforge
   Debian RedHat OpenBSD
   ocamlforge inria
   Big
   OCaml
   Project
   Primary
   mechanism for
   users to get
   software
   Packaged
   with apt, rpm,
   ports, etc
   No centralised
   distribution or version
   control
   in 2009
   

   View Slide

7. opam-repository
   OS Distributions
   Big
   OCaml
   Project
   OCaml
   compiler
   sourceforge
   Debian RedHat OpenBSD
   ocamlforge inria
   Big
   OCaml
   Project
   lwt
   re extlib
   cryptokit
   async pxp
   xen
   in 2012
   opam
   

   View Slide

8. opam-repository
   OS Distributions
   Big
   OCaml
   Project
   OCaml
   compiler
   sourceforge
   Debian RedHat OpenBSD
   ocamlforge inria
   Big
   OCaml
   Project
   lwt
   re extlib
   cryptokit
   async pxp
   xen
   Centralised
   git repository of
   community code
   Chunk
   projects into
   shareable
   libraries
   in 2012
   opam
   

   View Slide

9. opam-repository
   OS Distributions
   Big
   OCaml
   Project
   OCaml
   compiler
   github
   Debian RedHat OpenBSD
   github
   Big
   OCaml
   Project
   lwt
   re extlib
   cryptokit
   async pxp
   dbm
   ocaml
   labltk
   camlp4
   xen
   github
   in 2014
   opam
   

   View Slide

10. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    in 2014
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Almost
    everyone shifts
    to GitHub for
    hosting
    Compiler
    starts moving
    packages into
    opam
    opam
    

    View Slide

11. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    in 2015
    opam
    

    View Slide

12. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    Lots more
    packaging work
    for OS distros
    opam used
    by most OCaml
    developers
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    in 2015
    opam
    opam CI
    tests many
    distributions on
    every PR
    

    View Slide

13. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    opam
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    Reason

    ML
    in 2017
    Package Managers
    npm
    yarn
    esy
    

    View Slide

14. opam-repository
    OS Distributions
    Big
    OCaml
    Project
    OCaml
    compiler
    github
    Debian RedHat OpenBSD
    github
    Big
    OCaml
    Project
    opam
    lwt
    re extlib
    cryptokit
    async pxp
    dbm
    ocaml
    labltk
    camlp4
    xen
    github
    Arch macOS
    Alpine
    OpenSUSE
    Windows
    FreeBSD
    ocamlbuild ppx
    Reason

    ML
    in 2017
    Package Managers
    npm
    yarn
    esy
    Now
    packaging
    gets complex!
    Different
    workflows
    Poor opam
    is being
    overloaded
    

    View Slide

15. A preponderance of
    wrapping
    

    View Slide

16. make
    ocaml
    dpkg
    apt
    manual process
    of wrapping OCaml
    packages
    fixed
    versions of
    OCaml and
    libraries
    OS distro
    packaging
    done by distro
    maintainers
    

    View Slide

17. make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    OS distro Dev Workflow
    

    View Slide

18. make
    ocaml
    dpkg
    apt opam
    ocaml
    source
    oriented
    developer
    workflow
    opam-repo
    ocamlbuild
    multiple
    versions of
    OCaml and
    packages
    wrapping
    “make” or
    “ocamlbuild”
    OS distro Dev Workflow
    distro maintainers take
    a snapshot of opam-repo
    regularly
    

    View Slide

19. make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    opam
    ocamlbuild
    opam-repo
    make
    esy
    npm
    OS distro Dev Workflow ReasonML
    ocaml
    

    View Slide

20. OS distro Dev Workflow ReasonML
    make
    ocaml
    dpkg
    apt opam
    ocaml
    opam-repo
    ocamlbuild
    “distros cant keep up
    with package volume!”
    - Debian maintainer
    “pinning 100 packages
    takes a long time!”
    - MirageOS developer
    opam
    ocamlbuild
    opam-repo
    make
    esy
    npm
    ocaml
    “opam needs lock files to
    match the npm workflow!”
    - ReasonML maintainer
    “everyone is complaining
    about different things!”
    - Anil Madhavapeddy
    

    View Slide

21. opam
    lib
    merlin
    lib
    debian redhat openbsd
    alpine arch macos
    npm brew
    esy
    lib
    ounit
    ppx
    mdx
    yarn
    cygwin
    wsl
    odoc
    format
    ocamlfind
    ocamlbuild
    SLOW
    ocaml
    

    View Slide

22. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve
    

    View Slide

23. The OCaml Platform
    Challenge #1:

    incrementally update tools without breaking any workflow
    Challenge #2:

    remove endless layers of tools wrapping tools
    

    Our answer: dune and opam
    

    - composable, declarative builds: specialised to OCaml
    - reduce choice: deprecate ocamlbuild, omake, make
    - far higher performance and reproducibility in return.
    in 2017
    in 2019
    

    View Slide

24. Rearranging
    responsibilities
    opam the tool has been overloaded!
    • We have two distinct styles of workflows:
    • libraries: collections of OCaml modules
    • projects: executables, data
    • shift build logic into new tool: dune
    • packaging is what opam continues to do
    

    View Slide

25. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    

    View Slide

26. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    

    View Slide

27. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.1.0
    install foo.1.0
    make bar.0.3
    install bar.0.3
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    

    View Slide

28. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.1.0
    install foo.1.0
    make bar.0.3
    install bar.0.3
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    

    View Slide

29. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.2.0
    install foo.2.0
    make bar.1.0
    install bar.1.0
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    

    View Slide

30. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam update foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    make foo.2.0
    install foo.2.0
    make bar.1.0
    install bar.1.0
    opam solver
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    install bin/foo
    install lib/ocaml/bar
    make; ocamlbuild; ocamlfind;
    ocamldep; ocamlc; ocamlopt
    

    View Slide

31. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    

    View Slide

32. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo
    

    View Slide

33. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo
    Packaging
    

    View Slide

34. foo.1.0 foo.2.0
    bar.0.2 bar.0.3 bar.1.0
    opam install foo
    ocaml.4.07.1
    bar.0.3
    foo.1.0
    opam solver
    ocaml.4.08.1
    bar.1.0
    foo.2.0
    opam source bar.0.3
    opam source foo.1.0
    dune build
    ocamldep
    ocamlc
    ocamlopt
    bin/foo
    Packaging
    Build
    

    View Slide

35. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve
    

    View Slide

36. Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    opam install
    version
    constraints
    multiple valid
    choices of
    versions
    Role: Library Author
    

    View Slide

37. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    git clone
    dune install
    Role: End User
    

    View Slide

38. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    binary is the
    only public output
    of projects
    git clone
    dune install
    every library
    needed is in
    source tree
    Role: End User
    dubbed a
    “duniverse”
    

    View Slide

39. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    Role: Project Maintainer
    

    View Slide

40. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    duniverse init
    uses opam solver
    to pull libraries into
    project
    library upgrade is
    per project
    Role: Project Maintainer
    

    View Slide

41. Projects
    unison.1.2.0
    tezos.2.3.0
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    Role: Distribution Maintainer
    

    View Slide

42. Projects
    unison.1.2.0
    tezos.2.3.0
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    debian can solve
    for a coinstallable
    library / project set
    dune can generate
    deb package
    metadata
    Role: Distribution Maintainer
    

    View Slide

43. Projects
    unison.git
    tezos.git
    /lib
    /foo
    /src
    /bar
    /lib
    /foo
    /src
    /bar
    /bin
    unison
    /bin
    tezos
    Libraries
    foo.1.0
    foo.2.0
    bar.0.2
    bar.0.3
    bar.1.0
    esy / npm can
    vendor multiple
    versions of libraries
    dune artefact
    cache does sharing
    Role: Web developer
    

    View Slide

44. merlin
    ocamlformat
    mdx ppx
    ounit
    dune
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/
    

    View Slide

45. opam
    merlin
    ocamlformat
    mdx ppx
    ounit
    dune
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/
    

    View Slide

46. opam
    merlin
    debian
    redhat
    *bsd
    alpine arch
    macos
    ocamlformat
    mdx ppx
    ounit
    generate
    upstreamable
    OS packages
    dune
    cygwin
    wsl
    ubuntu
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/
    

    View Slide

47. opam
    merlin
    debian
    redhat
    *bsd
    alpine arch
    macos
    npm esy
    ocamlformat
    mdx ppx
    ounit
    generate
    upstreamable
    OS packages
    dune
    cygwin
    wsl
    generate package
    manager
    metadata
    ubuntu
    multiple versions
    of libraries
    composable build
    specifications
    bun
    cohttp
    ctypes infer
    coq
    flow mirageos
    f*
    zarith
    libs/ app/ tools/
    yarn
    

    View Slide

48. User archetypes
    • “Library author”
    - Reusable ML logic, not directly executable
    - OCaml algorithms, protocols, data structures, numeric methods, …
    • “End user”
    - Wants to use an application written in OCaml
    - Has little patience to learn tools, wants to solve problems!
    • “Distribution maintainer”
    - Works on maintaining a Linux distro like Debian or RedHat
    - Needs to make OCaml apps interoperate with their own policies
    • “Web/mobile/unikernel developer”
    - Write in a frontend syntax like ReasonML
    - Cross-compile to targets like JavaScript, wasm, iOS, Android
    end-to-end workflows to improve
    

    View Slide

49. Next Steps: Library Author
    • Continue to use opam as you do today!
    • 1.2 -> 2.0 switch - everyone has switched now. Regular
    point releases of 2.0.x to cover portability, workflow fixes,
    sandboxing improvements.
    
    • New maintainer Raja Boujbel has cut recent releases,
    along with Louis Gesbert and David Allsopp.
    
    • Plans for opam 2.1 and 2.2 are: a new solver based on
    Z3, Windows support, integrated depext.
    

    View Slide

50. Next Steps: Library Author
    0
    2000
    4000
    6000
    8000
    10000
    12000
    01/01/12 01/01/13 01/01/14 01/01/15 01/01/16 01/01/17 01/01/18 01/01/19 01/01/20
    %s
    Time
    unique opam packages
    

    View Slide

51. Next Steps: Library Author
    • Existing CI system is popular and useful, but
    cumbersome to use
    • https://ci.ocaml.org/
    • Working on next generation CI to make life easier:
    • https://github.com/ocaml-ci/
    
    • https://ci.ocamllabs.io
    
    • Based on a DSL similar to dune rules, so can express
    complex workflows.
    
    • Term language that can work with CLI and web
    
    • Automate bug triage — revdep breakage
    
    • Easy to extend with Discourse, Slack, etc
    

    View Slide

52. Next Steps: End User
    • Projects using duniverse now:
    • https://github.com/realworldocaml/book
    • https://github.com/avsm/duniverse
    • https://github.com/avsm/platform
    • Dune has been one of the fastest adopted
    Platform efforts since opam.
    • git clone; dune build
    

    View Slide

53. Next Steps: End User
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    building
    "duniverse"
    monorepo for
    Platform project
    mdx
    

    View Slide

54. Next Steps: End User
    • Coming soon in dune 2.0:

    dune artefact cache
    • https://github.com/ocaml/dune/pull/2493
    • Reliable per-rule caching of binary artefacts
    • Works locally first, eventually networking
    • Will eventually support reproducible builds
    throughout our stack, and therefore safe
    networked binary caches of OCaml code.
    

    View Slide

55. Next Steps: Project Maintainer
    • duniverse tool to bridge opam and dune
    https://github.com/avsm/duniverse
    • currently supports:
    • duniverse init: turn opam packages into git refs
    • duniverse pull: fetch git refs to local tree
    • coming soon:
    • duniverse update: merge new opam packages
    • duniverse install: add new opam packages
    

    View Slide

56. Next Steps: Project Maintainer
    • Do you have to use dune to get these benefits?
    

    View Slide

57. Next Steps: Project Maintainer
    Yes
    

    View Slide

58. Next Steps: Project Maintainer
    • Do you have to use dune to get these benefits?
    • Yes, but we maintain a transition remote:
    • https://github.com/dune-universe/opam-overlays
    • duniverse prototype tool automates inserting
    overrides.
    • We will provide a formal specification of dune
    files, so other implementations can be built.
    

    View Slide

59. Next Steps: Project Maintainer
    commit 632bc2eed00704bbbcdb5531d709c8d3f0cb0939
    Author: Thomas Gazagnaire
    Date: Wed Mar 28 15:49:16 2012 +0200
    Initial commit
    commit db943c24fe3ed5cec4dd432b1a01c66602d5f2f3
    Author: Xavier Leroy
    Date: Sun Jul 24 19:04:21 2016 +0200
    Add depexts for CentOS et al
    ~4 years
    opam-repo
    

    View Slide

60. Next Steps: Project Maintainer
    commit 632bc2eed00704bbbcdb5531d709c8d3f0cb0939
    Author: Thomas Gazagnaire
    Date: Wed Mar 28 15:49:16 2012 +0200
    Initial commit
    commit db943c24fe3ed5cec4dd432b1a01c66602d5f2f3
    Author: Xavier Leroy
    Date: Sun Jul 24 19:04:21 2016 +0200
    Add depexts for CentOS et al
    ~4 years
    commit 93b5d9bdb9c25d1ea86a96a9c9fce93ecebd5c6d
    Author: Jeremie Dimino
    Date: Sun Oct 30 19:07:53 2016 +0000
    initial import
    opam-repo
    dune
    

    View Slide

61. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage, why3, frama-c)
    • ocaml 4.08.x has early dune files
    

    View Slide

62. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage, why3, frama-c)
    • ocaml 4.08.x has early dune files
    (lang dune 1.10)
    ;; every metadata file is versioned
    

    View Slide

63. Next Steps: Project Maintainer
    • But seriously, do not switch until ready.
    • Transitions for big projects can take years.
    • dune project is ready and willing to help users
    transition (e.g. coq, mirage)
    • ocaml 4.08.x has early dune files
    CHANGES in one year:
    1.2.0: polling mode for build, private modules
    1.3.0: robust merlin file generation, colors on Windows!
    1.4.0: automatic code formatter targets, menhir rules
    1.5.0: @check target for rapid type checking, cross compilation for ocamlfind
    1.6.0: more flexible directory layouts for embedding in other projects
    1.7.0: improved optimisation for cmx, install metadata, formally specified cycle detection
    1.8.0: code promotion improvements, better c++ support, merlin on Windows
    1.9.0: library variants and default implementations, coq support
    1.10.0: generate opam metadata, future syntax shims, compile-time selection of os/arch
    1.11.0: dune init, cinaps support, build version embedding, inline test improvements
    

    View Slide

64. Next Steps: Distribution Maintainer
    • Our Linux distribution friends need more
    assistance!
    • Historically they have had good tooling to
    bridge, but now need better opam/dune
    integration.
    • Challenging new project in 2020 is Deb/RPM
    generation from duniverses.
    

    View Slide

65. Next Steps: Distribution Maintainer
    • Vision: multiple coinstallable and buildable
    trees that represent all of Debian’s or
    CentOS’s OCaml code in one git tree.
    • This is incredibly essential for the future of
    OCaml development.
    • A single dune build can test tens of millions
    of lines of OCaml code with multicore,
    flambda, algebraic effects and other new
    features in OCaml.
    

    View Slide

66. Next Steps: web/mobile/unikernel
    developer
    • duniverses make cross compilation to exotic
    targets extremely viable.
    • see talk on MirageOS 4 later today!
    • wrapping just a build system like dune frees
    esy / npm to do their own packaging
    • huge ecosystem potential of JavaScript,
    wasm, iOS/Android, unikernel, unlocked by
    OCaml
    

    View Slide

67. The OCaml Platform
    opam LTS (2.0.5) LGPLv2
    dune LTS (1.11.1) MIT DCO
    merlin stable (3.3.2) MIT
    utop stable (2.4.1) BSD3
    ocp-indent stable (1.7.0) LGPLv2
    mdx devel (1.4.0) ISC
    bun devel (0.3.3) MIT
    odoc devel (1.4.1) ISC
    dune-release devel (1.3.2) ISC
    ocamlformat devel (0.11.0) MIT CLA
    in 2019
    discuss.ocaml.org
    

    View Slide

68. The OCaml Platform
    in 2019…
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    mdx
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    Closer
    integration
    between tools
    dune-release
    building
    

    View Slide

69. Contact Lead Maintainers
    opam LTS (2.0.5) Raja Boujbel, Louis Gesbert
    dune LTS (1.11.1) Jeremie Dimino, Rudi Grinberg
    merlin stable (3.3.2) Fred Bour, Thomas Refis
    utop stable (2.4.1) Jeremie Dimino
    ocp-indent stable (1.7.0) Louis Gesbert
    mdx devel (1.4.0) Thomas Gazagnaire
    bun devel (0.3.3) Mindy Preston
    odoc devel (1.4.1) Anton Bachin, Jon Ludlam
    dune-release devel (1.3.2) Thomas Gazagnaire, Nathan Rebours
    ocamlformat devel (0.11.0) Josh Berdine, Guillaume Petiot
    in 2019
    discuss.ocaml.org
    

    View Slide

70. The OCaml Platform
    in 2019…
    opam dune
    merlin odoc
    ocamlformat
    ocp-indent
    utop
    bun
    publishing
    autoformatting
    editing documenting
    ppx_expect
    testing
    building
    "duniverse"
    monorepo for
    Platform
    project
    mdx
    

    View Slide

71. The OCaml Platform
    opam LTS (2.0.5) LGPLv2
    dune LTS (1.11.1) MIT DCO
    merlin stable (3.3.2) MIT
    utop stable (2.4.1) BSD3
    ocp-indent stable (1.7.0) LGPLv2
    mdx devel (1.4.0) ISC
    bun devel (0.3.3) MIT
    odoc devel (1.4.1) ISC
    dune-release devel (1.3.2) ISC
    ocamlformat devel (0.11.0) MIT CLA
    

    View Slide