OCaml community contributors and particularly Jeremie Dimino, Louis Gesbert, Raja Boujbel, Thomas Gazagnaire, Nathan Rebours for the content here (full list on slide 3) OCaml Workshop @ ICFP August 2019 Anil Madhavapeddy (speaker), Gemma Gordon, David Allsopp University of Cambridge Computer Laboratory
Allsopp, Leo Andres, Olivier Andrieu, Florian Angeletti, Edgar Aroutiounian, Nicolas Assouad, Tuncer Ayaz, Anton Bachin, Nicolas Ojeda Bar, Mathieu Barbin, Yotam Barnoy, Jochen Bartl, Jacob Bass, Romain Beauxis, Daniel Below, Matthias Andreas Benkard, Josh Berdine, Vincent Bernardoff, Vincent Berthoux, Francois Bobot, Richard Bonichon, Nick Borden, Mehdi Bouaziz, Raja Boujbel, Frédéric Bour, Daniel Buenzli, Elliott Cable, Romain Calascibetta, Benjamin Canou, Hezekiah Carty, Pierre Chambart, Amir Chaudhry, Kaustuv Chaudhuri, Xavier Clerc, Raphael Cornet, Roberto di Cosmo, Simon Cruanes, Tim Cuthbertson, Milo Davis, Richard Davison, Enguerrand Decorne, Ian Denhardt, Kate Deplaix, Dani Dickstein, Jeremie Dimino, Stephen Dolan, Damien Doliguez, Matthieu Dubuget, Tim Dysinger, Spiros Eliopoulos, John Else, Mohamed Elsharnouby, Shon Feder, Fabrice Le Fessant, Jean- Christophe Filliatre, Jared Forsyth, Simon Fowler, Alain Frisch, Anders Peter Fugmann, Sylvain Le Gall, Jacques Garrigue, Thomas Gazagnaire, Louis Gesbert, Gaetan Gilbert, Geoff Gole, Gemma Gordon, Ivan Gotovchits, Rudi Grinberg, Michael Grunewald, Oliver Gu, Adrien Guatto, Xavier Guerin, Armaël Gueneau, Mads Hartmann, Andreas Hauptmann, Philipp Haselwarter Gregoire Henry, Hugo Heuzard Daniel Hillerström, Sebastien Hinderer, Juergen Hoetzel, Wilfred Hughes, Ronan Le Hy, Takayuki Imada, Keigo Imai, Riso Isrof, Vincent Jacques, Martin Jambon, Kevin Ji, David Kaloper-Mersinjak, Brandon Kase, Anton Kochkov, Tomasz Kolodziejski, Chis Konstad, Marek Kubica, Marc Lasson, Theo Laurent, Qi Li, Nandor Licker, Jon Ludlam, Thomas Leonard, Rodolphe Lepigre, Maxime Lesourd, Gabe Levi, Martin Lucina, Ximin Luo, Anil Madhavapeddy, Louis Mandel, Luc Marange, Claude Marche, Thierry Martinez, Laurent Mazare, Tim McGilchrist Hannes Mehnert, Perry E. Metzger, Wojciech Meyer, Etienne Millon, Ben Millwood, Samuel Mimram, Yaron Minsky, Cedric Le Moigne, Sebastien Mondet, Florent Monnier, Richard Mortier, Darin Morrison, Markus Mottl, Max Mouratov, Tony Narlock, Andre Nathan, Olivier Nicole, Jason Nielsen, Gregory Nisbet, Andrew Noyes, Leandro Ostera, Saswat Padhi, Egor Panfilov, Vasilis Papavasileiou, Guillaume Petiot, Bryan Phelps, Zach Ploskey, Lucas Pluvinage, Francois Pottier, Nicolas Pouillard, Mindy Preston, Bobby Priambodo, Dominic Price, Raphael Proust, Steve Purcell, Gabriel Radanne, Tahina Ramananandro, Daniel de Rauglaudre, Andrew Ray, Nathan Rebours, Thomas Refis, Jan Rehders, Chris Reichert, Andrea Richiardi, Marshall Roch, Louis Roche, Mario Rodas, Leonid Rozenberg, Julien Sagot, Gabriel Scherer, Bernhard Schommer, David Scott, Pavel Senchanka, Marcello Seri, David Sheets, Mark Shinwell, Zach Shipko, KC Sivaramakrishnan, Magnus Skjegstad, Ryan Slade, Romain Slootmaekers, Roma Sokolov, Anurag Soni, Benjamin San Souci, Patrick Stapfer, Jason Staten, Quentin Stievenart, Alessandro Strada, Sander Spies, Yury Sulsky, Thibault Suzanne, Willy Tan, Dario Teixeira, Rijnard van Tonder Edwin Torok, Christophe Troestler, Martin Trojer, Frederic Tuong, Pierre Vandenhove, Raman Varabets, Ramana Venkata, Ricky Vetter, Jordan Walke, Liang Wang, Philippe Wang, Marc Weber, Pierre Weis, Sebastian Wiesner, Leo White, John Whitington, Maverick Woo, Justin Wood, Hao Wu, Hongchang Wu, Jeremy Yallop, Syohei Yoshida, Jianxin Zhao, aalekseyev, art-w, asmanur, Axiles, chrismamo1, copy, dkim, Fourchaux, gerdstolpmann, Kakadu, madroach, mfp, mwweissmann, nberth, ncihnegn, nv-vn, Nyavlys, orbitz, polazarus, prometheansacrifice, pszilagyi, pveber, sliquister, tategakibunko, tddsg, torkve, whitequark, ygrek, zogg
OpenBSD ocamlforge inria Big OCaml Project Primary mechanism for users to get software Packaged with apt, rpm, ports, etc No centralised distribution or version control in 2009
RedHat OpenBSD ocamlforge inria Big OCaml Project lwt re extlib cryptokit async pxp xen Centralised git repository of community code Chunk projects into shareable libraries in 2012 opam
RedHat OpenBSD in 2014 github Big OCaml Project lwt re extlib cryptokit async pxp dbm ocaml labltk camlp4 xen github Almost everyone shifts to GitHub for hosting Compiler starts moving packages into opam opam
RedHat OpenBSD github Big OCaml Project lwt re extlib cryptokit async pxp dbm ocaml labltk camlp4 xen github Arch macOS Alpine Lots more packaging work for OS distros opam used by most OCaml developers OpenSUSE Windows FreeBSD ocamlbuild ppx in 2015 opam opam CI tests many distributions on every PR
opam-repo ocamlbuild multiple versions of OCaml and packages wrapping “make” or “ocamlbuild” OS distro Dev Workflow distro maintainers take a snapshot of opam-repo regularly
ocaml opam-repo ocamlbuild “distros cant keep up with package volume!” - Debian maintainer “pinning 100 packages takes a long time!” - MirageOS developer opam ocamlbuild opam-repo make esy npm ocaml “opam needs lock files to match the npm workflow!” - ReasonML maintainer “everyone is complaining about different things!” - Anil Madhavapeddy
directly executable - OCaml algorithms, protocols, data structures, numeric methods, … • “End user” - Wants to use an application written in OCaml - Has little patience to learn tools, wants to solve problems! • “Distribution maintainer” - Works on maintaining a Linux distro like Debian or RedHat - Needs to make OCaml apps interoperate with their own policies • “Web/mobile/unikernel developer” - Write in a frontend syntax like ReasonML - Cross-compile to targets like JavaScript, wasm, iOS, Android end-to-end workflows to improve
any workflow Challenge #2: remove endless layers of tools wrapping tools Our answer: dune and opam - composable, declarative builds: specialised to OCaml - reduce choice: deprecate ocamlbuild, omake, make - far higher performance and reproducibility in return. in 2017 in 2019
have two distinct styles of workflows: • libraries: collections of OCaml modules • projects: executables, data • shift build logic into new tool: dune • packaging is what opam continues to do
directly executable - OCaml algorithms, protocols, data structures, numeric methods, … • “End user” - Wants to use an application written in OCaml - Has little patience to learn tools, wants to solve problems! • “Distribution maintainer” - Works on maintaining a Linux distro like Debian or RedHat - Needs to make OCaml apps interoperate with their own policies • “Web/mobile/unikernel developer” - Write in a frontend syntax like ReasonML - Cross-compile to targets like JavaScript, wasm, iOS, Android end-to-end workflows to improve
/bar /bin unison /bin tezos binary is the only public output of projects git clone dune install every library needed is in source tree Role: End User dubbed a “duniverse”
/bar /bin unison /bin tezos Libraries foo.1.0 foo.2.0 bar.0.2 bar.0.3 bar.1.0 debian can solve for a coinstallable library / project set dune can generate deb package metadata Role: Distribution Maintainer
directly executable - OCaml algorithms, protocols, data structures, numeric methods, … • “End user” - Wants to use an application written in OCaml - Has little patience to learn tools, wants to solve problems! • “Distribution maintainer” - Works on maintaining a Linux distro like Debian or RedHat - Needs to make OCaml apps interoperate with their own policies • “Web/mobile/unikernel developer” - Write in a frontend syntax like ReasonML - Cross-compile to targets like JavaScript, wasm, iOS, Android end-to-end workflows to improve
you do today! • 1.2 -> 2.0 switch - everyone has switched now. Regular point releases of 2.0.x to cover portability, workflow fixes, sandboxing improvements. • New maintainer Raja Boujbel has cut recent releases, along with Louis Gesbert and David Allsopp. • Plans for opam 2.1 and 2.2 are: a new solver based on Z3, Windows support, integrated depext.
and useful, but cumbersome to use • https://ci.ocaml.org/ • Working on next generation CI to make life easier: • https://github.com/ocaml-ci/ • https://ci.ocamllabs.io • Based on a DSL similar to dune rules, so can express complex workflows. • Term language that can work with CLI and web • Automate bug triage — revdep breakage • Easy to extend with Discourse, Slack, etc
https://github.com/realworldocaml/book • https://github.com/avsm/duniverse • https://github.com/avsm/platform • Dune has been one of the fastest adopted Platform efforts since opam. • git clone; dune build
dune artefact cache • https://github.com/ocaml/dune/pull/2493 • Reliable per-rule caching of binary artefacts • Works locally first, eventually networking • Will eventually support reproducible builds throughout our stack, and therefore safe networked binary caches of OCaml code.
dune to get these benefits? • Yes, but we maintain a transition remote: • https://github.com/dune-universe/opam-overlays • duniverse prototype tool automates inserting overrides. • We will provide a formal specification of dune files, so other implementations can be built.
until ready. • Transitions for big projects can take years. • dune project is ready and willing to help users transition (e.g. coq, mirage, why3, frama-c) • ocaml 4.08.x has early dune files
until ready. • Transitions for big projects can take years. • dune project is ready and willing to help users transition (e.g. coq, mirage, why3, frama-c) • ocaml 4.08.x has early dune files (lang dune 1.10) ;; every metadata file is versioned
until ready. • Transitions for big projects can take years. • dune project is ready and willing to help users transition (e.g. coq, mirage) • ocaml 4.08.x has early dune files CHANGES in one year: 1.2.0: polling mode for build, private modules 1.3.0: robust merlin file generation, colors on Windows! 1.4.0: automatic code formatter targets, menhir rules 1.5.0: @check target for rapid type checking, cross compilation for ocamlfind 1.6.0: more flexible directory layouts for embedding in other projects 1.7.0: improved optimisation for cmx, install metadata, formally specified cycle detection 1.8.0: code promotion improvements, better c++ support, merlin on Windows 1.9.0: library variants and default implementations, coq support 1.10.0: generate opam metadata, future syntax shims, compile-time selection of os/arch 1.11.0: dune init, cinaps support, build version embedding, inline test improvements
more assistance! • Historically they have had good tooling to bridge, but now need better opam/dune integration. • Challenging new project in 2020 is Deb/RPM generation from duniverses.
trees that represent all of Debian’s or CentOS’s OCaml code in one git tree. • This is incredibly essential for the future of OCaml development. • A single dune build can test tens of millions of lines of OCaml code with multicore, flambda, algebraic effects and other new features in OCaml.
exotic targets extremely viable. • see talk on MirageOS 4 later today! • wrapping just a build system like dune frees esy / npm to do their own packaging • huge ecosystem potential of JavaScript, wasm, iOS/Android, unikernel, unlocked by OCaml
avsm
kubotak
yuhta28
utgwkk
vrallev
mierune
minodriven
masashi
e869120
michaelsimons
olivergierke
track3jyo
charity
mojombo
afnizarnur
smashingmag
brad_frost
akmur
eileencodes
frogandcode
samanthasiow
stephaniewalter
wjessup
jnunemaker
lara
