Tweet
Tweet

Slide 1

Slide 1 text

The OCaml Platform 1.0 With many contributions from the OCaml community contributors
 (full list on slide 3) OCaml Workshop, St Louis September 2018 Anil Madhavapeddy, Gemma Gordon
 University of Cambridge Computer Laboratory

Slide 2

Slide 2 text

The OCaml Platform The OCaml Platform combines the OCaml compiler toolchain with a coherent set of tools for build, documentation, testing and IDE integration. The project is a collaborative effort across the OCaml community, tied together by the OCaml Labs group in Cambridge, and OCamlPro in Paris, and the whole OCaml community.

Slide 3

Slide 3 text

Platform Contributors: 250+ Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad, Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below, Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean- Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu, Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi, Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan, Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou, Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo, Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg, Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci, Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets, Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w, asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn, Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg

Slide 4

Slide 4 text

The OCaml Platform opam publishing building dune

Slide 5

Slide 5 text

The OCaml Platform opam publishing several opam 2 betas to gather feedback and improve workflows in 2017… Feb 2017: opam 2~beta1 Mar 2017: opam2~beta2 May 2017: opam2~beta3 June 2017: opam 1.2.0 deprecated Aug 2017: opam 2~beta4 Nov 2017: opam 2~beta5

Slide 6

Slide 6 text

The OCaml Platform opam publishing building several opam 2 betas to gather feedback and improve workflows jbuilder was initially released to solve build, later renamed to dune in 2017… Feb 2017: opam 2~beta1 Mar 2017: opam2~beta2 May 2017: opam2~beta3 June 2017: opam 1.2.0 deprecated Aug 2017: opam 2~beta4 Nov 2017: opam 2~beta5 Mar 2017: jbuilder beta1 Apr 2017: jbuilder beta2-9 July 2017: opam builds using jbuilder Aug 2017: beta10-13, more developers Sep 2017: in top 10 opam packages Nov 2017: jbuilder beta16 dune

Slide 7

Slide 7 text

The OCaml Platform publishing building in 2018… opam dune

Slide 8

Slide 8 text

The OCaml Platform publishing building Jan 2018: opam 2~rc1 May 2018: critical build issue in camlp5 May 2018: opam 2~rc2 (sandboxing) July 2018: opam 2~rc3 Aug 2018: opam 2~rc4 Sep 2018: opam 2.0.0 stable released! Sep 2018: opam-repo using 2.0 format. in 2018… opam dune opam 2.0.0 released with v1.2-v2.0 migration and secure sandboxing

Slide 9

Slide 9 text

The OCaml Platform publishing building Jan 2018: jbuilder beta17 Feb 2018: jbuilder beta18 Mar 2018: jbuilder beta19 Apr 2018: jbuilder beta20 May 2018: promoted to ocaml/ org July 2018: dune 1.0 stable released Aug 2018: dune 1.1 stable Sep 2018: dune 1.2 stable opam 2.0.0 released with v1.2-v2.0 migration and secure sandboxing dune 1.x stable released and widely adopted by community in 2018… opam dune Jan 2018: opam 2~rc1 May 2018: critical build issue in camlp5 May 2018: opam 2~rc2 (sandboxing) July 2018: opam 2~rc3 Aug 2018: opam 2~rc4 Sep 2018: opam 2.0.0 stable released! Sep 2018: opam-repo using 2.0 format.

Slide 10

Slide 10 text

opam 2 • stable release in Sep 2018 after 3.5 years of development! • new developer workflows: local switches, lockfiles, richer package description language for more sophisticated constraints. • more portable with built-in solver and switch to OCaml compiler being a package, and foundations for full Windows support (coming in 2.1!). • more secure builds from source via secure OS sandboxing on macOS and Linux, so rogue code cannot compromise host machine. opam.ocaml.org

Slide 11

Slide 11 text

dune (formerly jbuilder) • fast adoption by the OCaml community, high velocity of releases, and growing maintainer list. • overall has been faster, more batteries included and more portable than previous solutions. • compositional builds unlock flexible switching between opam and dune workflows • lint and promotion mechanisms assist with code hygiene, and assist with interface migrations. • carefully limited computation model ensures consistency of build rules across large codebases. dune.readthedocs.org

Slide 12

Slide 12 text

stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1)))

Slide 13

Slide 13 text

stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1))) dune-project file defines semantic version of dune build but we use a feature introduced in a later version of dune

Slide 14

Slide 14 text

stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) File "dune", line 4, characters 13-31: 4 | (preprocess (staged_pps -arg1))) ^^^^^^^^^^^^^^^^^^ Error: 'staged_pps' is only available since version 1.1 of the dune language (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1))) helpful error messages with hint on how to fix the problem

Slide 15

Slide 15 text

What’s in the
 OCaml Platform? opam and dune form the stable foundations

Slide 16

Slide 16 text

The OCaml Platform publishing building opam dune ocaml

Slide 17

Slide 17 text

The OCaml Platform publishing building opam dune ocaml shims and disaggregated compiler libraries num graphics dbm uchar result bytes

Slide 18

Slide 18 text

The OCaml Platform publishing building num graphics dbm uchar result bytes opam dune ocaml merlin odoc ocamlformat dune-release ocp-indent mdx utop supported tooling shims and disaggregated compiler libraries

Slide 19

Slide 19 text

What’s in the
 OCaml Platform 1.0? a set of stable OCaml tools, all compiled, tested and released together

Slide 20

Slide 20 text

The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.3.0) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (1.0.1) ISC ocamlformat devel (0.7) MIT CLA

Slide 21

Slide 21 text

The OCaml Platform Linux Debian 9 x86_64, ppc64le, arm32v7, arm64 Alpine 3 x86_64, arm64 CentOS 7 x86_64 OracleLinux 7 x86_64 OpenSUSE 42.3 x86_64 Fedora 28 x86_64 Ubuntu 18.04 x86_64, arm64 Ubuntu 16.04 x86_64, arm64 macOS High Sierra + x86_64 FreeBSD 11.1 x86_64 OpenBSD 6.3 x86_64 Windows Windows 10 x86_64, x86 https://gitlab.com/ocaml-platform/ci/pipelines

Slide 22

Slide 22 text

The OCaml Platform Linux Debian 9 Deb repository Alpine 3 Apk packages CentOS 7 RPMs OracleLinux 7 RPMs OpenSUSE 42.3 RPMs Fedora 28 RPMs Ubuntu 18.04 PPA Ubuntu 16.04 PPA macOS High Sierra + Homebrew FreeBSD 11.1 Binaries OpenBSD 6.3 Ports / Packages Windows Windows 10 Binaries / MSI https://gitlab.com/ocaml-platform/ci/pipelines

Slide 23

Slide 23 text

The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.2.1) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (0.3.0) ISC ocamlformat devel (0.7) MIT CLA

Slide 24

Slide 24 text

The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.2.1) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (0.3.0) ISC ocamlformat devel (0.7) MIT CLA Eventually, these tools will merge into one CLI driver }

Slide 25

Slide 25 text

How is the
 OCaml Platform built and released?

Slide 26

Slide 26 text

OCaml Platform releases • Can offer longer term support for some OCaml compiler versions, depending on demand. • Can offer precompiled host tools from a monorepo that is bootstrapped quickly. • Monorepo provides a place to test platform tools with each other simultaneously. • Bleeding edge versions of the compiler can still have useful tooling (i.e. avoid ppx breakage).

Slide 27

Slide 27 text

The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc

Slide 28

Slide 28 text

The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc dune-release

Slide 29

Slide 29 text

The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc Converted to Git refs opam.git#v2.0.0 dune.git#v1.2.1 merlin.git#3.1.0 …etc dune-release duniverse git-lock New tool to bridge opam and dune
 dubbed “duniverse” https://github.com/avsm/duniverse Runs heuristics to convert opam packages to a single dune build.

Slide 30

Slide 30 text

The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx Converted to Git refs opam.git#v2.0.0 dune.git#v1.2.1 merlin.git#3.1.0 …etc dune-release duniverse
 git-lock duniverse
 git-pull

Slide 31

Slide 31 text

The Duniverse Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx $ find ocaml_modules -name \*.ml -o -name \*.mli | \
 xargs wc -l
 1148464 total

Slide 32

Slide 32 text

The Duniverse Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx $ find ocaml_modules -name \*.ml -o -name \*.mli | \
 xargs wc -l
 1148464 total All code tracked in one repository Updated from opam metadata Everything must be ported to Dune
 For transitions, we have https://github.com/dune-universe

Slide 33

Slide 33 text

The Duniverse Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx

Slide 34

Slide 34 text

The Duniverse Test multiplatform versioned from monorepo 4.07.0-1 4.07.1-3 4.08.0-1~dev Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli CI make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx

Slide 35

Slide 35 text

The Duniverse Test multiplatform versioned from monorepo 4.07.0-1 4.07.1-3 4.08.0-1~dev Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli CI make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx OS package managers Homebrew tap Ubuntu PPA Arch

Slide 36

Slide 36 text

…the architecture of the OCaml Platform is therefore a standalone, versioned monorepo that bootstraps cleanly from a C compiler on all supported OS distributions. https://github.com/avsm/platform

Slide 37

Slide 37 text

merlin • Editor support is near complete now for all popular editors. VS Code and Atom have plugins that work out-of-the-box. • More backend features: • “polarity search” enabled type-based search • open refactoring cleans up code in the context of an open statement • type-driven record completion simplifies editing • Improved portability via Windows and FreeBSD support. Status: stable 3.1.0

Slide 38

Slide 38 text

utop • A modern top-level for OCaml with command completion and a nice terminal interface. • Has been quiet for the past few years, but now re- energised by moving to a new community maintainership:
 https://github.com/ocaml-community/utop • After being integrated into Platform, portability also improving and more consistent CI. Status: stable 2.2.0

Slide 39

Slide 39 text

…and tools in development for inclusion

Slide 40

Slide 40 text

ocamlformat • A tool to format OCaml source code consistently. • Aims for code legibility, exposing the high-level structure while remaining compact, and spotting syntactic gotchas like dangling if/else. • Been in rapid development all year, and can now format huge codebases (e.g. all of Jane Street’s internal code) • Works alongside ocp-indent, but the tools will merge. • Good example of a “new” Platform tool: liberal license, CLA/ DCO, community contributions, clear maintainership Status: development

Slide 41

Slide 41 text

odoc • Documentation generator, intended as a replacement for ocamldoc. • Still under development, and increasingly contributor-friendly. • Major advances this year: • support for Dune so the two ways to invoke it are now odig and dune build @doc • a unified Git repository that is easier to hack on. • Still todo before a release are a cleanup of the type-driven analysis (the “doc-ock” component) and full language coverage. Status: development

Slide 42

Slide 42 text

…and tools ready for inclusion

Slide 43

Slide 43 text

mdx • expect-based tests to execute code blocks within Markdown file. • Works for shell scripts, and OCaml fragments, including continuation support for breaking up large toplevels. Status: stable ```sh $ cat < hello\ > world\ > EOF hello world ```

Slide 44

Slide 44 text

mdx • expect-based tests to execute code blocks within Markdown file. • Dune supports promoting and show differences in tests outputs. Status: stable $ dune runtest ------ file.md ++++++ file.md.corrected File "file.md", line 23, characters 0-1: | |```sh -| $ for i in `seq 1 3`; do echo $i; done +| $ for i in `seq 1 4`; do echo $i; done | 1 | 2 | 3 +| 4 |```

Slide 45

Slide 45 text

mdx • expect-based tests to execute code blocks within Markdown file. • OCaml toplevels and fragments can be built and tested using standard Dune rules. Status: stable ```ocaml # print_endline "42" 42 ```

Slide 46

Slide 46 text

What’s left for the
 OCaml Platform 1.0 release in 2018?

Slide 47

Slide 47 text

Next steps • Define a contribution process for adding and removing packages from the Platform list. An “RFC” process. • Generate unified changelogs for easier upstream consumption. • Windows support in all tools (David Allsopp’s talk later). • The hard vendoring problem: where do the fixes go? Immediate next steps: I will post on discuss.ocaml.org with 4.07.1~beta1 in ~October Meanwhile, view the preview mono-repository at:
 https://github.com/avsm/platform
 If any tools are missing you think should be included, talk to me!

Slide 48

Slide 48 text

Contact Lead Maintainers opam LTS (2.0.0) Raja Boujbel, Louis Gesbert dune LTS (1.3.0) Jeremie Dimino, Rudi Grinberg merlin stable (3.1.0) Fred Bour, Thomas Refis utop stable (2.2.0) Jeremie Dimino ocp-indent stable (1.6.1) Louis Gesbert mdx stable (1.0.0) Thomas Gazagnaire bun devel (0.3.2) Mindy Preston odoc devel (1.2.0) Thomas Refis, Leo White, Anton Bachin dune-release devel (1.0.1) Thomas Gazagnaire ocamlformat devel (0.7) Josh Berdine, Hugo Heuzard