The OCaml Platform 1.0 (2018)

Introducing the architecture and design behind the forthcoming OCaml Platform 1.0

Transcript

1. The OCaml Platform 1.0 With many contributions from the OCaml community contributors
 (full list on slide 3) OCaml Workshop, St Louis September 2018 Anil Madhavapeddy, Gemma Gordon
 University of Cambridge Computer Laboratory
2. The OCaml Platform The OCaml Platform combines the OCaml compiler toolchain with a coherent set of tools for build, documentation, testing and IDE integration. The project is a collaborative effort across the OCaml community, tied together by the OCaml Labs group in Cambridge, and OCamlPro in Paris, and the whole OCaml community.
3. Platform Contributors: 250+ Akinori Abe, Ashish Agarwal, Ion Alberdi, David Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad, Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below, Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean- Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu, Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi, Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan, Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou, Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo, Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg, Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci, Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets, Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w, asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn, Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg
4. The OCaml Platform opam publishing building dune
5. The OCaml Platform opam publishing several opam 2 betas to gather feedback and improve workflows in 2017… Feb 2017: opam 2~beta1 Mar 2017: opam2~beta2 May 2017: opam2~beta3 June 2017: opam 1.2.0 deprecated Aug 2017: opam 2~beta4 Nov 2017: opam 2~beta5
6. The OCaml Platform opam publishing building several opam 2 betas to gather feedback and improve workflows jbuilder was initially released to solve build, later renamed to dune in 2017… Feb 2017: opam 2~beta1 Mar 2017: opam2~beta2 May 2017: opam2~beta3 June 2017: opam 1.2.0 deprecated Aug 2017: opam 2~beta4 Nov 2017: opam 2~beta5 Mar 2017: jbuilder beta1 Apr 2017: jbuilder beta2-9 July 2017: opam builds using jbuilder Aug 2017: beta10-13, more developers Sep 2017: in top 10 opam packages Nov 2017: jbuilder beta16 dune
7. The OCaml Platform publishing building in 2018… opam dune
8. The OCaml Platform publishing building Jan 2018: opam 2~rc1 May 2018: critical build issue in camlp5 May 2018: opam 2~rc2 (sandboxing) July 2018: opam 2~rc3 Aug 2018: opam 2~rc4 Sep 2018: opam 2.0.0 stable released! Sep 2018: opam-repo using 2.0 format. in 2018… opam dune opam 2.0.0 released with v1.2-v2.0 migration and secure sandboxing
9. The OCaml Platform publishing building Jan 2018: jbuilder beta17 Feb 2018: jbuilder beta18 Mar 2018: jbuilder beta19 Apr 2018: jbuilder beta20 May 2018: promoted to ocaml/ org July 2018: dune 1.0 stable released Aug 2018: dune 1.1 stable Sep 2018: dune 1.2 stable opam 2.0.0 released with v1.2-v2.0 migration and secure sandboxing dune 1.x stable released and widely adopted by community in 2018… opam dune Jan 2018: opam 2~rc1 May 2018: critical build issue in camlp5 May 2018: opam 2~rc2 (sandboxing) July 2018: opam 2~rc3 Aug 2018: opam 2~rc4 Sep 2018: opam 2.0.0 stable released! Sep 2018: opam-repo using 2.0 format.
10. opam 2 • stable release in Sep 2018 after 3.5 years of development! • new developer workflows: local switches, lockfiles, richer package description language for more sophisticated constraints. • more portable with built-in solver and switch to OCaml compiler being a package, and foundations for full Windows support (coming in 2.1!). • more secure builds from source via secure OS sandboxing on macOS and Linux, so rogue code cannot compromise host machine. opam.ocaml.org
11. dune (formerly jbuilder) • fast adoption by the OCaml community, high velocity of releases, and growing maintainer list. • overall has been faster, more batteries included and more portable than previous solutions. • compositional builds unlock flexible switching between opam and dune workflows • lint and promotion mechanisms assist with code hygiene, and assist with interface migrations. • carefully limited computation model ensures consistency of build rules across large codebases. dune.readthedocs.org
12. stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1)))
13. stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1))) dune-project file defines semantic version of dune build but we use a feature introduced in a later version of dune
14. stable versioning • Huge emphasis in both tools on versioned metadata so that newly released tools do not break existing builds. (lang dune 1.0) (name mirage-types) File "dune", line 4, characters 13-31: 4 | (preprocess (staged_pps -arg1))) ^^^^^^^^^^^^^^^^^^ Error: 'staged_pps' is only available since version 1.1 of the dune language (library (name mirage_ppx) (modules mirage_ppx) (preprocess (staged_pps -arg1))) helpful error messages with hint on how to fix the problem
15. What’s in the
 OCaml Platform? opam and dune form the stable foundations
16. The OCaml Platform publishing building opam dune ocaml
17. The OCaml Platform publishing building opam dune ocaml shims and disaggregated compiler libraries num graphics dbm uchar result bytes
18. The OCaml Platform publishing building num graphics dbm uchar result bytes opam dune ocaml merlin odoc ocamlformat dune-release ocp-indent mdx utop supported tooling shims and disaggregated compiler libraries
19. What’s in the
 OCaml Platform 1.0? a set of stable OCaml tools, all compiled, tested and released together
20. The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.3.0) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (1.0.1) ISC ocamlformat devel (0.7) MIT CLA
21. The OCaml Platform Linux Debian 9 x86_64, ppc64le, arm32v7, arm64 Alpine 3 x86_64, arm64 CentOS 7 x86_64 OracleLinux 7 x86_64 OpenSUSE 42.3 x86_64 Fedora 28 x86_64 Ubuntu 18.04 x86_64, arm64 Ubuntu 16.04 x86_64, arm64 macOS High Sierra + x86_64 FreeBSD 11.1 x86_64 OpenBSD 6.3 x86_64 Windows Windows 10 x86_64, x86 https://gitlab.com/ocaml-platform/ci/pipelines
22. The OCaml Platform Linux Debian 9 Deb repository Alpine 3 Apk packages CentOS 7 RPMs OracleLinux 7 RPMs OpenSUSE 42.3 RPMs Fedora 28 RPMs Ubuntu 18.04 PPA Ubuntu 16.04 PPA macOS High Sierra + Homebrew FreeBSD 11.1 Binaries OpenBSD 6.3 Ports / Packages Windows Windows 10 Binaries / MSI https://gitlab.com/ocaml-platform/ci/pipelines
23. The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.2.1) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (0.3.0) ISC ocamlformat devel (0.7) MIT CLA
24. The OCaml Platform opam LTS (2.0.0) LGPLv2 dune LTS (1.2.1) MIT DCO merlin stable (3.1.0) MIT utop stable (2.2.0) BSD3 ocp-indent stable (1.6.1) LGPLv2 mdx stable (1.0.0) ISC bun devel (0.3.2) MIT odoc devel (1.2.0) ISC dune-release devel (0.3.0) ISC ocamlformat devel (0.7) MIT CLA Eventually, these tools will merge into one CLI driver }
25. How is the
 OCaml Platform built and released?
26. OCaml Platform releases • Can offer longer term support for some OCaml compiler versions, depending on demand. • Can offer precompiled host tools from a monorepo that is bootstrapped quickly. • Monorepo provides a place to test platform tools with each other simultaneously. • Bleeding edge versions of the compiler can still have useful tooling (i.e. avoid ppx breakage).
27. The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc
28. The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc dune-release
29. The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc Converted to Git refs opam.git#v2.0.0 dune.git#v1.2.1 merlin.git#3.1.0 …etc dune-release duniverse git-lock New tool to bridge opam and dune
 dubbed “duniverse” https://github.com/avsm/duniverse Runs heuristics to convert opam packages to a single dune build.
30. The Duniverse Multiple GitHub repositories github.com/ocaml/opam github.com/ocaml/dune github.com/ocaml/merlin …etc Published to opam opam 2.0.0 dune 1.2.1 merlin 3.1.0 …etc Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx Converted to Git refs opam.git#v2.0.0 dune.git#v1.2.1 merlin.git#3.1.0 …etc dune-release duniverse
 git-lock duniverse
 git-pull
31. The Duniverse Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx $ find ocaml_modules -name \*.ml -o -name \*.mli | \
 xargs wc -l
 1148464 total
32. The Duniverse Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx $ find ocaml_modules -name \*.ml -o -name \*.mli | \
 xargs wc -l
 1148464 total All code tracked in one repository Updated from opam metadata Everything must be ported to Dune
 For transitions, we have https://github.com/dune-universe
33. The Duniverse Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx
34. The Duniverse Test multiplatform versioned from monorepo 4.07.0-1 4.07.1-3 4.08.0-1~dev Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli CI make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx
35. The Duniverse Test multiplatform versioned from monorepo 4.07.0-1 4.07.1-3 4.08.0-1~dev Build binaries [bootstrap ocaml] [bootstrap dune] dune build @cli CI make Convert to Dune monorepo github.com/avsm/platform / ocaml_modules/ merlin / ocaml_modules / odoc / ocaml_modules / mdx OS package managers Homebrew tap Ubuntu PPA Arch
36. …the architecture of the OCaml Platform is therefore a standalone, versioned monorepo that bootstraps cleanly from a C compiler on all supported OS distributions. https://github.com/avsm/platform
37. merlin • Editor support is near complete now for all popular editors. VS Code and Atom have plugins that work out-of-the-box. • More backend features: • “polarity search” enabled type-based search • open refactoring cleans up code in the context of an open statement • type-driven record completion simplifies editing • Improved portability via Windows and FreeBSD support. Status: stable 3.1.0
38. utop • A modern top-level for OCaml with command completion and a nice terminal interface. • Has been quiet for the past few years, but now re- energised by moving to a new community maintainership:
 https://github.com/ocaml-community/utop • After being integrated into Platform, portability also improving and more consistent CI. Status: stable 2.2.0
39. …and tools in development for inclusion
40. ocamlformat • A tool to format OCaml source code consistently. • Aims for code legibility, exposing the high-level structure while remaining compact, and spotting syntactic gotchas like dangling if/else. • Been in rapid development all year, and can now format huge codebases (e.g. all of Jane Street’s internal code) • Works alongside ocp-indent, but the tools will merge. • Good example of a “new” Platform tool: liberal license, CLA/ DCO, community contributions, clear maintainership Status: development
41. odoc • Documentation generator, intended as a replacement for ocamldoc. • Still under development, and increasingly contributor-friendly. • Major advances this year: • support for Dune so the two ways to invoke it are now odig and dune build @doc • a unified Git repository that is easier to hack on. • Still todo before a release are a cleanup of the type-driven analysis (the “doc-ock” component) and full language coverage. Status: development
42. …and tools ready for inclusion
43. mdx • expect-based tests to execute code blocks within Markdown file. • Works for shell scripts, and OCaml fragments, including continuation support for breaking up large toplevels. Status: stable ```sh $ cat <<EOF \ > hello\ > world\ > EOF hello world ```
44. mdx • expect-based tests to execute code blocks within Markdown file. • Dune supports promoting and show differences in tests outputs. Status: stable $ dune runtest ------ file.md ++++++ file.md.corrected File "file.md", line 23, characters 0-1: | |```sh -| $ for i in `seq 1 3`; do echo $i; done +| $ for i in `seq 1 4`; do echo $i; done | 1 | 2 | 3 +| 4 |```
45. mdx • expect-based tests to execute code blocks within Markdown file. • OCaml toplevels and fragments can be built and tested using standard Dune rules. Status: stable ```ocaml # print_endline "42" 42 ```
46. What’s left for the
 OCaml Platform 1.0 release in 2018?
47. Next steps • Define a contribution process for adding and removing packages from the Platform list. An “RFC” process. • Generate unified changelogs for easier upstream consumption. • Windows support in all tools (David Allsopp’s talk later). • The hard vendoring problem: where do the fixes go? Immediate next steps: I will post on discuss.ocaml.org with 4.07.1~beta1 in ~October Meanwhile, view the preview mono-repository at:
 https://github.com/avsm/platform
 If any tools are missing you think should be included, talk to me!
48. Contact Lead Maintainers opam LTS (2.0.0) Raja Boujbel, Louis Gesbert dune LTS (1.3.0) Jeremie Dimino, Rudi Grinberg merlin stable (3.1.0) Fred Bour, Thomas Refis utop stable (2.2.0) Jeremie Dimino ocp-indent stable (1.6.1) Louis Gesbert mdx stable (1.0.0) Thomas Gazagnaire bun devel (0.3.2) Mindy Preston odoc devel (1.2.0) Thomas Refis, Leo White, Anton Bachin dune-release devel (1.0.1) Thomas Gazagnaire ocamlformat devel (0.7) Josh Berdine, Hugo Heuzard