Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The OCaml Platform 1.0 (2018)

The OCaml Platform 1.0 (2018)

Introducing the architecture and design behind the forthcoming OCaml Platform 1.0

Anil Madhavapeddy

September 27, 2018
Tweet

More Decks by Anil Madhavapeddy

Other Decks in Programming

Transcript

  1. The OCaml
    Platform 1.0
    With many contributions from the OCaml community contributors

    (full list on slide 3)
    OCaml Workshop, St Louis
    September 2018
    Anil Madhavapeddy, Gemma Gordon

    University of Cambridge Computer Laboratory

    View Slide

  2. The OCaml Platform
    The OCaml Platform combines the OCaml
    compiler toolchain with a coherent set of tools for
    build, documentation, testing and IDE integration.
    The project is a collaborative effort across the OCaml
    community, tied together by the OCaml Labs group in
    Cambridge, and OCamlPro in Paris, and the whole
    OCaml community.

    View Slide

  3. Platform Contributors: 250+
    Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad,
    Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below,
    Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi
    Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre
    Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo
    Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien
    Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean-
    Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas
    Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu,
    Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard
    Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent
    Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek
    Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi,
    Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim
    McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le
    Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan,
    Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou,
    Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo,
    Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan
    Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg,
    Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach
    Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci,
    Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario
    Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets,
    Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John
    Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w,
    asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn,
    Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg

    View Slide

  4. The OCaml Platform
    opam
    publishing building
    dune

    View Slide

  5. The OCaml Platform
    opam
    publishing
    several opam 2 betas
    to gather feedback
    and improve workflows
    in 2017…
    Feb 2017: opam 2~beta1
    Mar 2017: opam2~beta2
    May 2017: opam2~beta3

    June 2017: opam 1.2.0 deprecated
    Aug 2017: opam 2~beta4

    Nov 2017: opam 2~beta5

    View Slide

  6. The OCaml Platform
    opam
    publishing building
    several opam 2 betas
    to gather feedback
    and improve workflows
    jbuilder was initially
    released to solve build,
    later renamed to dune
    in 2017…
    Feb 2017: opam 2~beta1
    Mar 2017: opam2~beta2
    May 2017: opam2~beta3

    June 2017: opam 1.2.0 deprecated
    Aug 2017: opam 2~beta4

    Nov 2017: opam 2~beta5
    Mar 2017: jbuilder beta1
    Apr 2017: jbuilder beta2-9
    July 2017: opam builds using jbuilder

    Aug 2017: beta10-13, more developers
    Sep 2017: in top 10 opam packages

    Nov 2017: jbuilder beta16
    dune

    View Slide

  7. The OCaml Platform
    publishing building
    in 2018…
    opam dune

    View Slide

  8. The OCaml Platform
    publishing building
    Jan 2018: opam 2~rc1
    May 2018: critical build issue in camlp5

    May 2018: opam 2~rc2 (sandboxing)
    July 2018: opam 2~rc3
    Aug 2018: opam 2~rc4

    Sep 2018: opam 2.0.0 stable released!

    Sep 2018: opam-repo using 2.0 format.
    in 2018…
    opam dune
    opam 2.0.0 released
    with v1.2-v2.0 migration
    and secure sandboxing

    View Slide

  9. The OCaml Platform
    publishing building
    Jan 2018: jbuilder beta17
    Feb 2018: jbuilder beta18
    Mar 2018: jbuilder beta19

    Apr 2018: jbuilder beta20
    May 2018: promoted to ocaml/ org

    July 2018: dune 1.0 stable released

    Aug 2018: dune 1.1 stable
    Sep 2018: dune 1.2 stable
    opam 2.0.0 released
    with v1.2-v2.0 migration
    and secure sandboxing
    dune 1.x stable
    released and widely
    adopted by community
    in 2018…
    opam dune
    Jan 2018: opam 2~rc1
    May 2018: critical build issue in camlp5

    May 2018: opam 2~rc2 (sandboxing)
    July 2018: opam 2~rc3
    Aug 2018: opam 2~rc4

    Sep 2018: opam 2.0.0 stable released!

    Sep 2018: opam-repo using 2.0 format.

    View Slide

  10. opam 2
    • stable release in Sep 2018 after 3.5 years of development!

    • new developer workflows: local switches, lockfiles, richer
    package description language for more sophisticated
    constraints.

    • more portable with built-in solver and switch to OCaml
    compiler being a package, and foundations for full Windows
    support (coming in 2.1!).

    • more secure builds from source via secure OS sandboxing
    on macOS and Linux, so rogue code cannot compromise
    host machine.
    opam.ocaml.org

    View Slide

  11. dune (formerly jbuilder)
    • fast adoption by the OCaml community, high velocity of releases,
    and growing maintainer list.

    • overall has been faster, more batteries included and more
    portable than previous solutions.

    • compositional builds unlock flexible switching between opam and
    dune workflows

    • lint and promotion mechanisms assist with code hygiene, and
    assist with interface migrations.
    • carefully limited computation model ensures consistency of build
    rules across large codebases.
    dune.readthedocs.org

    View Slide

  12. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))

    View Slide

  13. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))
    dune-project file defines semantic
    version of dune build
    but we use a feature introduced in
    a later version of dune

    View Slide

  14. stable versioning
    • Huge emphasis in both tools on versioned metadata so
    that newly released tools do not break existing builds.
    (lang dune 1.0)
    (name mirage-types)
    File "dune", line 4, characters 13-31:
    4 | (preprocess (staged_pps -arg1)))
    ^^^^^^^^^^^^^^^^^^
    Error: 'staged_pps' is only available since version 1.1 of
    the dune language
    (library
    (name mirage_ppx)
    (modules mirage_ppx)
    (preprocess (staged_pps -arg1)))
    helpful error messages with hint
    on how to fix the problem

    View Slide

  15. What’s in the

    OCaml Platform?
    opam and dune form
    the stable foundations

    View Slide

  16. The OCaml Platform
    publishing building
    opam dune
    ocaml

    View Slide

  17. The OCaml Platform
    publishing building
    opam dune
    ocaml
    shims and
    disaggregated
    compiler libraries
    num
    graphics
    dbm
    uchar
    result
    bytes

    View Slide

  18. The OCaml Platform
    publishing building
    num
    graphics
    dbm
    uchar
    result
    bytes
    opam dune
    ocaml
    merlin
    odoc
    ocamlformat
    dune-release
    ocp-indent
    mdx
    utop
    supported
    tooling
    shims and
    disaggregated
    compiler libraries

    View Slide

  19. What’s in the

    OCaml Platform 1.0?
    a set of stable OCaml tools,
    all compiled, tested and
    released together

    View Slide

  20. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.3.0) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (1.0.1) ISC
    ocamlformat devel (0.7) MIT CLA

    View Slide

  21. The OCaml Platform
    Linux Debian 9 x86_64, ppc64le, arm32v7, arm64
    Alpine 3 x86_64, arm64
    CentOS 7 x86_64
    OracleLinux 7 x86_64
    OpenSUSE 42.3 x86_64
    Fedora 28 x86_64
    Ubuntu 18.04 x86_64, arm64
    Ubuntu 16.04 x86_64, arm64
    macOS High Sierra + x86_64
    FreeBSD 11.1 x86_64
    OpenBSD 6.3 x86_64
    Windows Windows 10 x86_64, x86
    https://gitlab.com/ocaml-platform/ci/pipelines

    View Slide

  22. The OCaml Platform
    Linux Debian 9 Deb repository
    Alpine 3 Apk packages
    CentOS 7 RPMs
    OracleLinux 7 RPMs
    OpenSUSE 42.3 RPMs
    Fedora 28 RPMs
    Ubuntu 18.04 PPA
    Ubuntu 16.04 PPA
    macOS High Sierra + Homebrew
    FreeBSD 11.1 Binaries
    OpenBSD 6.3 Ports / Packages
    Windows Windows 10 Binaries / MSI
    https://gitlab.com/ocaml-platform/ci/pipelines

    View Slide

  23. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.2.1) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (0.3.0) ISC
    ocamlformat devel (0.7) MIT CLA

    View Slide

  24. The OCaml Platform
    opam LTS (2.0.0) LGPLv2
    dune LTS (1.2.1) MIT DCO
    merlin stable (3.1.0) MIT
    utop stable (2.2.0) BSD3
    ocp-indent stable (1.6.1) LGPLv2
    mdx stable (1.0.0) ISC
    bun devel (0.3.2) MIT
    odoc devel (1.2.0) ISC
    dune-release devel (0.3.0) ISC
    ocamlformat devel (0.7) MIT CLA
    Eventually,
    these tools will
    merge into one
    CLI driver
    }

    View Slide

  25. How is the

    OCaml Platform
    built and released?

    View Slide

  26. OCaml Platform releases
    • Can offer longer term support for some OCaml compiler
    versions, depending on demand.

    • Can offer precompiled host tools from a monorepo that
    is bootstrapped quickly.

    • Monorepo provides a place to test platform tools with
    each other simultaneously.

    • Bleeding edge versions of the compiler can still have
    useful tooling (i.e. avoid ppx breakage).

    View Slide

  27. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc

    View Slide

  28. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    dune-release

    View Slide

  29. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    Converted to Git refs
    opam.git#v2.0.0
    dune.git#v1.2.1
    merlin.git#3.1.0
    …etc
    dune-release
    duniverse
    git-lock
    New tool to bridge opam and dune

    dubbed “duniverse”
    https://github.com/avsm/duniverse
    Runs heuristics to convert opam
    packages to a single dune build.

    View Slide

  30. The Duniverse
    Multiple GitHub repositories
    github.com/ocaml/opam
    github.com/ocaml/dune
    github.com/ocaml/merlin
    …etc
    Published to opam
    opam 2.0.0
    dune 1.2.1
    merlin 3.1.0
    …etc
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    Converted to Git refs
    opam.git#v2.0.0
    dune.git#v1.2.1
    merlin.git#3.1.0
    …etc
    dune-release
    duniverse

    git-lock
    duniverse

    git-pull

    View Slide

  31. The Duniverse
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    $ find ocaml_modules -name \*.ml -o -name \*.mli | \

    xargs wc -l

    1148464 total

    View Slide

  32. The Duniverse
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    $ find ocaml_modules -name \*.ml -o -name \*.mli | \

    xargs wc -l

    1148464 total
    All code tracked in one repository
    Updated from opam metadata
    Everything must be ported to Dune

    For transitions, we have https://github.com/dune-universe

    View Slide

  33. The Duniverse
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx

    View Slide

  34. The Duniverse
    Test multiplatform
    versioned from monorepo
    4.07.0-1
    4.07.1-3
    4.08.0-1~dev
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    CI
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx

    View Slide

  35. The Duniverse
    Test multiplatform
    versioned from monorepo
    4.07.0-1
    4.07.1-3
    4.08.0-1~dev
    Build binaries
    [bootstrap ocaml]
    [bootstrap dune]
    dune build @cli
    CI
    make
    Convert to Dune monorepo
    github.com/avsm/platform
    / ocaml_modules/ merlin
    / ocaml_modules / odoc
    / ocaml_modules / mdx
    OS package managers
    Homebrew tap
    Ubuntu PPA
    Arch

    View Slide

  36. …the architecture of the OCaml Platform is
    therefore a standalone, versioned
    monorepo that bootstraps cleanly from a C
    compiler on all supported OS distributions.
    https://github.com/avsm/platform

    View Slide

  37. merlin
    • Editor support is near complete now for all popular editors. VS
    Code and Atom have plugins that work out-of-the-box.

    • More backend features:

    • “polarity search” enabled type-based search

    • open refactoring cleans up code in the context of an open
    statement

    • type-driven record completion simplifies editing

    • Improved portability via Windows and FreeBSD support.
    Status: stable 3.1.0

    View Slide

  38. utop
    • A modern top-level for OCaml with command completion
    and a nice terminal interface.

    • Has been quiet for the past few years, but now re-
    energised by moving to a new community maintainership:

    https://github.com/ocaml-community/utop

    • After being integrated into Platform, portability also
    improving and more consistent CI.
    Status: stable 2.2.0

    View Slide

  39. …and tools in
    development for
    inclusion

    View Slide

  40. ocamlformat
    • A tool to format OCaml source code consistently.

    • Aims for code legibility, exposing the high-level structure
    while remaining compact, and spotting syntactic gotchas like
    dangling if/else.

    • Been in rapid development all year, and can now format huge
    codebases (e.g. all of Jane Street’s internal code)

    • Works alongside ocp-indent, but the tools will merge.

    • Good example of a “new” Platform tool: liberal license, CLA/
    DCO, community contributions, clear maintainership
    Status: development

    View Slide

  41. odoc
    • Documentation generator, intended as a replacement for ocamldoc.

    • Still under development, and increasingly contributor-friendly.

    • Major advances this year:

    • support for Dune so the two ways to invoke it are now odig and
    dune build @doc

    • a unified Git repository that is easier to hack on.

    • Still todo before a release are a cleanup of the type-driven analysis
    (the “doc-ock” component) and full language coverage.
    Status: development

    View Slide

  42. …and tools ready for
    inclusion

    View Slide

  43. mdx
    • expect-based tests to execute code blocks within
    Markdown file.

    • Works for shell scripts, and OCaml fragments, including
    continuation support for breaking up large toplevels.
    Status: stable
    ```sh
    $ cat <> hello\
    > world\
    > EOF
    hello
    world
    ```

    View Slide

  44. mdx
    • expect-based tests to execute code blocks within
    Markdown file.

    • Dune supports promoting and show differences in tests
    outputs.
    Status: stable
    $ dune runtest
    ------ file.md
    ++++++ file.md.corrected
    File "file.md", line 23, characters 0-1:
    |
    |```sh
    -| $ for i in `seq 1 3`; do echo $i; done
    +| $ for i in `seq 1 4`; do echo $i; done
    | 1
    | 2
    | 3
    +| 4
    |```

    View Slide

  45. mdx
    • expect-based tests to execute code blocks within
    Markdown file.

    • OCaml toplevels and fragments can be built and tested
    using standard Dune rules.
    Status: stable
    ```ocaml
    # print_endline "42"
    42
    ```

    View Slide

  46. What’s left for the

    OCaml Platform 1.0
    release in 2018?

    View Slide

  47. Next steps
    • Define a contribution process for adding and removing
    packages from the Platform list. An “RFC” process.

    • Generate unified changelogs for easier upstream
    consumption.

    • Windows support in all tools (David Allsopp’s talk later).

    • The hard vendoring problem: where do the fixes go?
    Immediate next steps:
    I will post on discuss.ocaml.org with 4.07.1~beta1 in ~October
    Meanwhile, view the preview mono-repository at:

    https://github.com/avsm/platform

    If any tools are missing you think should be included, talk to me!

    View Slide

  48. Contact Lead Maintainers
    opam LTS (2.0.0) Raja Boujbel, Louis Gesbert
    dune LTS (1.3.0) Jeremie Dimino, Rudi Grinberg
    merlin stable (3.1.0) Fred Bour, Thomas Refis
    utop stable (2.2.0) Jeremie Dimino
    ocp-indent stable (1.6.1) Louis Gesbert
    mdx stable (1.0.0) Thomas Gazagnaire
    bun devel (0.3.2) Mindy Preston
    odoc devel (1.2.0) Thomas Refis, Leo White, Anton Bachin
    dune-release devel (1.0.1) Thomas Gazagnaire
    ocamlformat devel (0.7) Josh Berdine, Hugo Heuzard

    View Slide