The OCaml Platform 1.0 (2018)

Introducing the architecture and design behind the forthcoming OCaml Platform 1.0

Transcript

1. The OCaml
   Platform 1.0
   With many contributions from the OCaml community contributors

   (full list on slide 3)
   OCaml Workshop, St Louis
   September 2018
   Anil Madhavapeddy, Gemma Gordon

   University of Cambridge Computer Laboratory
   

   View Slide

2. The OCaml Platform
   The OCaml Platform combines the OCaml
   compiler toolchain with a coherent set of tools for
   build, documentation, testing and IDE integration.
   The project is a collaborative effort across the OCaml
   community, tied together by the OCaml Labs group in
   Cambridge, and OCamlPro in Paris, and the whole
   OCaml community.
   

   View Slide

3. Platform Contributors: 250+
   Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad,
   Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below,
   Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi
   Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre
   Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo
   Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien
   Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean-
   Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas
   Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu,
   Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard
   Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent
   Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek
   Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi,
   Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim
   McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le
   Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan,
   Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou,
   Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo,
   Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan
   Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg,
   Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach
   Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci,
   Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario
   Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets,
   Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John
   Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w,
   asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn,
   Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg
   

   View Slide

4. The OCaml Platform
   opam
   publishing building
   dune
   

   View Slide

5. The OCaml Platform
   opam
   publishing
   several opam 2 betas
   to gather feedback
   and improve workflows
   in 2017…
   Feb 2017: opam 2~beta1
   Mar 2017: opam2~beta2
   May 2017: opam2~beta3
   
   June 2017: opam 1.2.0 deprecated
   Aug 2017: opam 2~beta4
   
   Nov 2017: opam 2~beta5
   

   View Slide

6. The OCaml Platform
   opam
   publishing building
   several opam 2 betas
   to gather feedback
   and improve workflows
   jbuilder was initially
   released to solve build,
   later renamed to dune
   in 2017…
   Feb 2017: opam 2~beta1
   Mar 2017: opam2~beta2
   May 2017: opam2~beta3
   
   June 2017: opam 1.2.0 deprecated
   Aug 2017: opam 2~beta4
   
   Nov 2017: opam 2~beta5
   Mar 2017: jbuilder beta1
   Apr 2017: jbuilder beta2-9
   July 2017: opam builds using jbuilder
   
   Aug 2017: beta10-13, more developers
   Sep 2017: in top 10 opam packages
   
   Nov 2017: jbuilder beta16
   dune
   

   View Slide

7. The OCaml Platform
   publishing building
   in 2018…
   opam dune
   

   View Slide

8. The OCaml Platform
   publishing building
   Jan 2018: opam 2~rc1
   May 2018: critical build issue in camlp5
   
   May 2018: opam 2~rc2 (sandboxing)
   July 2018: opam 2~rc3
   Aug 2018: opam 2~rc4
   
   Sep 2018: opam 2.0.0 stable released!
   
   Sep 2018: opam-repo using 2.0 format.
   in 2018…
   opam dune
   opam 2.0.0 released
   with v1.2-v2.0 migration
   and secure sandboxing
   

   View Slide

9. The OCaml Platform
   publishing building
   Jan 2018: jbuilder beta17
   Feb 2018: jbuilder beta18
   Mar 2018: jbuilder beta19
   
   Apr 2018: jbuilder beta20
   May 2018: promoted to ocaml/ org
   
   July 2018: dune 1.0 stable released
   
   Aug 2018: dune 1.1 stable
   Sep 2018: dune 1.2 stable
   opam 2.0.0 released
   with v1.2-v2.0 migration
   and secure sandboxing
   dune 1.x stable
   released and widely
   adopted by community
   in 2018…
   opam dune
   Jan 2018: opam 2~rc1
   May 2018: critical build issue in camlp5
   
   May 2018: opam 2~rc2 (sandboxing)
   July 2018: opam 2~rc3
   Aug 2018: opam 2~rc4
   
   Sep 2018: opam 2.0.0 stable released!
   
   Sep 2018: opam-repo using 2.0 format.
   

   View Slide

10. opam 2
    • stable release in Sep 2018 after 3.5 years of development!
    
    • new developer workflows: local switches, lockfiles, richer
    package description language for more sophisticated
    constraints.
    
    • more portable with built-in solver and switch to OCaml
    compiler being a package, and foundations for full Windows
    support (coming in 2.1!).
    
    • more secure builds from source via secure OS sandboxing
    on macOS and Linux, so rogue code cannot compromise
    host machine.
    opam.ocaml.org
    

    View Slide

11. dune (formerly jbuilder)
    • fast adoption by the OCaml community, high velocity of releases,
    and growing maintainer list.
    
    • overall has been faster, more batteries included and more
    portable than previous solutions.
    
    • compositional builds unlock flexible switching between opam and
    dune workflows
    
    • lint and promotion mechanisms assist with code hygiene, and
    assist with interface migrations.
    • carefully limited computation model ensures consistency of build
    rules across large codebases.
    dune.readthedocs.org
    

    View Slide

12. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))
    

    View Slide

13. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))
    dune-project file defines semantic
    version of dune build
    but we use a feature introduced in
    a later version of dune
    

    View Slide

14. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    File "dune", line 4, characters 13-31:
    4 | (preprocess (staged_pps -arg1)))
    ^^^^^^^^^^^^^^^^^^
    Error: 'staged_pps' is only available since version 1.1 of
    the dune language
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))
    helpful error messages with hint
    on how to fix the problem
    

    View Slide

15. What’s in the

    OCaml Platform?
    opam and dune form
    the stable foundations
    

    View Slide

16. The OCaml Platform
    publishing building
    opam dune
    ocaml
    

    View Slide

17. The OCaml Platform
    publishing building
    opam dune
    ocaml
    shims and
    disaggregated
    compiler libraries
    num
    graphics
    dbm
    uchar
    result
    bytes
    

    View Slide

18. The OCaml Platform
    publishing building
    num
    graphics
    dbm
    uchar
    result
    bytes
    opam dune
    ocaml
    merlin
    odoc
    ocamlformat
    dune-release
    ocp-indent
    mdx
    utop
    supported
    tooling
    shims and
    disaggregated
    compiler libraries
    

    View Slide

19. What’s in the

    OCaml Platform 1.0?
    a set of stable OCaml tools,
    all compiled, tested and
    released together
    

    View Slide

20. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.3.0) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (1.0.1) ISC
    ocamlformat devel (0.7) MIT CLA
    

    View Slide

21. The OCaml Platform
    Linux Debian 9 x86_64, ppc64le, arm32v7, arm64
    Alpine 3 x86_64, arm64
    CentOS 7 x86_64
    OracleLinux 7 x86_64
    OpenSUSE 42.3 x86_64
    Fedora 28 x86_64
    Ubuntu 18.04 x86_64, arm64
    Ubuntu 16.04 x86_64, arm64
    macOS High Sierra + x86_64
    FreeBSD 11.1 x86_64
    OpenBSD 6.3 x86_64
    Windows Windows 10 x86_64, x86
    https://gitlab.com/ocaml-platform/ci/pipelines
    

    View Slide

22. The OCaml Platform
    Linux Debian 9 Deb repository
    Alpine 3 Apk packages
    CentOS 7 RPMs
    OracleLinux 7 RPMs
    OpenSUSE 42.3 RPMs
    Fedora 28 RPMs
    Ubuntu 18.04 PPA
    Ubuntu 16.04 PPA
    macOS High Sierra + Homebrew
    FreeBSD 11.1 Binaries
    OpenBSD 6.3 Ports / Packages
    Windows Windows 10 Binaries / MSI
    https://gitlab.com/ocaml-platform/ci/pipelines
    

    View Slide

23. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.2.1) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (0.3.0) ISC
    ocamlformat devel (0.7) MIT CLA
    

    View Slide

24. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.2.1) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (0.3.0) ISC
    ocamlformat devel (0.7) MIT CLA
    Eventually,
    these tools will
    merge into one
    CLI driver
    }
    

    View Slide

25. How is the

    OCaml Platform
    built and released?
    

    View Slide

26. OCaml Platform releases
    • Can offer longer term support for some OCaml compiler
    versions, depending on demand.
    
    • Can offer precompiled host tools from a monorepo that
    is bootstrapped quickly.
    
    • Monorepo provides a place to test platform tools with
    each other simultaneously.
    
    • Bleeding edge versions of the compiler can still have
    useful tooling (i.e. avoid ppx breakage).
    

    View Slide

27. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    

    View Slide

28. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    dune-release
    

    View Slide

29. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    Converted to Git refs
    opam.git#v2.0.0
    dune.git#v1.2.1
    merlin.git#3.1.0
    …etc
    dune-release
    duniverse
    git-lock
    New tool to bridge opam and dune

    dubbed “duniverse”
    https://github.com/avsm/duniverse
    Runs heuristics to convert opam
    packages to a single dune build.
    

    View Slide

30. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    Converted to Git refs
    opam.git#v2.0.0
    dune.git#v1.2.1
    merlin.git#3.1.0
    …etc
    dune-release
    duniverse

    git-lock
    duniverse

    git-pull
    

    View Slide

31. The Duniverse
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    $ find ocaml_modules -name \*.ml -o -name \*.mli | \

    xargs wc -l

    1148464 total
    

    View Slide

32. The Duniverse
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    $ find ocaml_modules -name \*.ml -o -name \*.mli | \

    xargs wc -l

    1148464 total
    All code tracked in one repository
    Updated from opam metadata
    Everything must be ported to Dune

    For transitions, we have https://github.com/dune-universe
    

    View Slide

33. The Duniverse
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    

    View Slide

34. The Duniverse
    Test multiplatform
    versioned from monorepo
    4.07.0-1
    4.07.1-3
    4.08.0-1~dev
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    CI
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    

    View Slide

35. The Duniverse
    Test multiplatform
    versioned from monorepo
    4.07.0-1
    4.07.1-3
    4.08.0-1~dev
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    CI
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    OS package managers
    Homebrew tap
    Ubuntu PPA
    Arch
    

    View Slide

36. …the architecture of the OCaml Platform is
    therefore a standalone, versioned
    monorepo that bootstraps cleanly from a C
    compiler on all supported OS distributions.
    https://github.com/avsm/platform
    

    View Slide

37. merlin
    • Editor support is near complete now for all popular editors. VS
    Code and Atom have plugins that work out-of-the-box.
    
    • More backend features:
    
    • “polarity search” enabled type-based search
    
    • open refactoring cleans up code in the context of an open
    statement
    
    • type-driven record completion simplifies editing
    
    • Improved portability via Windows and FreeBSD support.
    Status: stable 3.1.0
    

    View Slide

38. utop
    • A modern top-level for OCaml with command completion
    and a nice terminal interface.
    
    • Has been quiet for the past few years, but now re-
    energised by moving to a new community maintainership:

    https://github.com/ocaml-community/utop
    
    • After being integrated into Platform, portability also
    improving and more consistent CI.
    Status: stable 2.2.0
    

    View Slide

39. …and tools in
    development for
    inclusion
    

    View Slide

40. ocamlformat
    • A tool to format OCaml source code consistently.
    
    • Aims for code legibility, exposing the high-level structure
    while remaining compact, and spotting syntactic gotchas like
    dangling if/else.
    
    • Been in rapid development all year, and can now format huge
    codebases (e.g. all of Jane Street’s internal code)
    
    • Works alongside ocp-indent, but the tools will merge.
    
    • Good example of a “new” Platform tool: liberal license, CLA/
    DCO, community contributions, clear maintainership
    Status: development
    

    View Slide

41. odoc
    • Documentation generator, intended as a replacement for ocamldoc.
    
    • Still under development, and increasingly contributor-friendly.
    
    • Major advances this year:
    
    • support for Dune so the two ways to invoke it are now odig and
    dune build @doc
    
    • a unified Git repository that is easier to hack on.
    
    • Still todo before a release are a cleanup of the type-driven analysis
    (the “doc-ock” component) and full language coverage.
    Status: development
    

    View Slide

42. …and tools ready for
    inclusion
    

    View Slide

43. mdx
    • expect-based tests to execute code blocks within
    Markdown file.
    
    • Works for shell scripts, and OCaml fragments, including
    continuation support for breaking up large toplevels.
    Status: stable
    ```sh
    $ cat <> hello\
    > world\
    > EOF
    hello
    world
    ```
    

    View Slide

44. mdx
    • expect-based tests to execute code blocks within
    Markdown file.
    
    • Dune supports promoting and show differences in tests
    outputs.
    Status: stable
    $ dune runtest
    ------ file.md
    ++++++ file.md.corrected
    File "file.md", line 23, characters 0-1:
    |
    |```sh
    -| $ for i in `seq 1 3`; do echo $i; done
    +| $ for i in `seq 1 4`; do echo $i; done
    | 1
    | 2
    | 3
    +| 4
    |```
    

    View Slide

45. mdx
    • expect-based tests to execute code blocks within
    Markdown file.
    
    • OCaml toplevels and fragments can be built and tested
    using standard Dune rules.
    Status: stable
    ```ocaml
    # print_endline "42"
    42
    ```
    

    View Slide

46. What’s left for the

    OCaml Platform 1.0
    release in 2018?
    

    View Slide

47. Next steps
    • Define a contribution process for adding and removing
    packages from the Platform list. An “RFC” process.
    
    • Generate unified changelogs for easier upstream
    consumption.
    
    • Windows support in all tools (David Allsopp’s talk later).
    
    • The hard vendoring problem: where do the fixes go?
    Immediate next steps:
    I will post on discuss.ocaml.org with 4.07.1~beta1 in ~October
    Meanwhile, view the preview mono-repository at:

    https://github.com/avsm/platform

    If any tools are missing you think should be included, talk to me!
    

    View Slide

48. Contact Lead Maintainers
    opam LTS (2.0.0) Raja Boujbel, Louis Gesbert
    dune LTS (1.3.0) Jeremie Dimino, Rudi Grinberg
    merlin stable (3.1.0) Fred Bour, Thomas Refis
    utop stable (2.2.0) Jeremie Dimino
    ocp-indent stable (1.6.1) Louis Gesbert
    mdx stable (1.0.0) Thomas Gazagnaire
    bun devel (0.3.2) Mindy Preston
    odoc devel (1.2.0) Thomas Refis, Leo White, Anton Bachin
    dune-release devel (1.0.1) Thomas Gazagnaire
    ocamlformat devel (0.7) Josh Berdine, Hugo Heuzard
    

    View Slide