Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OCaml Platform 2020

OCaml Platform 2020

Keynote at the OCaml Workshop 2020 (part of the ACM ICFP 2020 series). The talk describes the advances in the OCaml community around tooling and the processes for the construction of the OCaml Platform.

Anil Madhavapeddy

August 31, 2020
Tweet

More Decks by Anil Madhavapeddy

Other Decks in Technology

Transcript

  1. State of the

    OCaml Platform
    2020
    With many contributions from the OCaml community and maintainers.
    OCaml Workshop @ ICFP
    August 2020
    Anil Madhavapeddy,
    University of Cambridge Computer Laboratory

    OCaml Labs

    View Slide

  2. The OCaml Platform makes
    users productive with the
    OCaml language
    This talk covers:
    Integrated Development Environments
    Next Steps for the OCaml Platform
    Plans for 2020-2021

    View Slide

  3. Integrated
    Development
    Environments
    • Most of the OCaml developer experience is
    spent within an IDE.
    • Traditionally, OCaml has had good support
    for Emacs, and "ok" support for Vim.
    • Editor support is complex due to the
    number of tools and latency requirements.

    View Slide

  4. Integrated
    Development
    Environments
    Emacs
    Vim
    Atom
    Tuareg
    Vimrc
    Plugin
    Merlin
    OCamlformat
    Odoc

    View Slide

  5. Integrated
    Development
    Environments
    Emacs
    Vim
    Atom
    Merlin
    OCamlformat
    Odoc
    Language

    Server
    Protocol
    (LSP)


    Server

    View Slide

  6. Integrated
    Development
    Environments
    Emacs
    Vim
    Atom
    Language

    Server
    Protocol
    (LSP)


    Server
    New OCaml project:
    github.com/ocaml/ocaml-lsp
    Integrates pieces of merlin,
    odoc, and omd into a single
    binary, so works out of the box.
    Builds on previous efforts, but in
    pure OCaml.

    View Slide

  7. • LSP provides a sustainable basis for expanding IDE
    support for OCaml.
    • Single binary means that complexities of tooling can
    be hidden behind it.
    • We also developed a complete Visual Studio Code
    plugin:
    • VSCode is popular, extensible and cross-platform.

    • Install the OCaml Platform extension and off you go.

    • Supports all the features newcomers expect from a
    modern IDE.
    Integrated
    Development
    Environments

    View Slide

  8. Types on hover

    View Slide

  9. Autocomplete

    View Slide

  10. Destructing pattern matches

    View Slide

  11. Project sandboxing

    View Slide

  12. • Try it out now by installing the "OCaml Platform" plugin
    in the VSCode Marketplace.

    • 100% free and open-source!
    Integrated
    Development
    Environments
    The plugin will transition to the "ocaml" organisation on GitHub.
    Next: how do we accept and maintain such packages?

    View Slide

  13. • For the first time, OCaml 4.11.0 was released
    simultaneously with support for popular tools.

    • Combined efforts of Florian Angeletti (OCaml release),
    Kate Deplaix (opam) and David Allsopp (OCaml core),
    and all the developers of the tools.

    • See ocaml/opam-repository#6539

    • Requires an extended release cycle for OCaml, with a
    new alpha phase before the more public beta.
    The OCaml Platform:
    Synchronous Release

    View Slide

  14. The OCaml Platform
    ocamlfind packaging 1.8.1 MIT sustain
    opam-publish packaging 2.0.2 LGPLv2 active
    opam packaging 2.0.7 LGPLv2 active
    dune-release packaging 1.4.0 ISC active
    odoc documenting 1.5.1 ISC incubate
    ocamlbuild building 0.14.0 LGPLv2 sustain
    dune building 2.7.0 MIT active
    merlin editing 3.3.7 MIT active
    utop editing 2.6.0 BSD3 active
    ocp-indent editing 1.8.1 LGPLv2 sustain
    ocamlformat editing 0.15.0 MIT incubate
    lsp-server editing 1.0.0 ISC incubate
    mdx testing 1.7.0 ISC incubate
    bun testing 0.3.3 MIT incubate
    omp ast 2.0.0 LGPLv2 sustain
    ppxlib ast 0.16.0 MIT active

    View Slide

  15. • Incubate
    • Projects that fill a gap in the ecosystem, or perform an existing function differently.

    • Not yet ready for mainstream release and have unreliable backwards compatibility.

    • Active
    • The day-to-day workhorse projects, with strong backwards compatibility.

    • May be adding major features, but not radically changing their workflows.

    • Sustain
    • No major new features added, but updated for OCaml releases.

    • Continue to use these, but be aware that there may be active alternatives.

    • Can be an extremely important part of the ecosystem and in this mode indefinitely.

    • Deprecate
    • Being pushed out of the OCaml distribution and platform.

    • There will be recommended alternatives, and projects should migrate.
    The OCaml Platform:
    Classification

    View Slide

  16. The OCaml Platform
    odoc documenting 1.5.1 ISC incubate
    ocamlformat editing 0.15.0 MIT incubate
    lsp-server editing 1.0.0 ISC incubate
    mdx testing 1.7.0 ISC incubate
    bun testing 0.3.3 MIT incubate
    dune-release packaging 1.4.0 ISC incubate
    opam packaging 2.0.7 LGPLv2 active
    opam-publish packaging 2.0.2 LGPLv2 active
    dune building 2.7.0 MIT active
    merlin editing 3.3.7 MIT active
    utop editing 2.6.0 BSD3 active
    ppxlib ast 0.16.0 MIT active
    ocamlfind packaging 1.8.1 MIT sustain
    ocamlbuild building 0.14.0 LGPLv2 sustain
    ocp-indent editing 1.8.1 LGPLv2 sustain
    omp ast 2.0.0 LGPLv2 sustain
    camlp4 ast 4.11.0 LGPLv2 deprecate
    oasis build 0.4.11 LGPLv2 deprecate

    View Slide

  17. • How do we decide what is in scope for the Platform?
    • Initially, just tools. Libraries will be supported as "transitive dependencies".

    (e.g. even ppx compiles into a binary driver)

    • Principles for acceptance:
    • Sharing: does this assist with the reuse and publication of OCaml code?
    • Development: does this improve the OCaml development process, ranging from
    onboarding new developers to maintaining a legacy codebase?
    • Evolution: is there a strategy for incremental change and backwards compatibility
    over several years?
    • Openness: is the source code liberally licensed, and are there any operational
    restrictions (such as online services) that the tool depends on?
    • Structural requirements:
    • Maintainer resources, establish a community need, demonstrate some adoption.
    The OCaml Platform:
    Scope

    View Slide

  18. The OCaml Platform: lifecycle
    Incubate Active
    Sustain
    Deprecate

    View Slide

  19. The OCaml Platform: lifecycle
    Incubate Active
    Sustain
    Deprecate
    Cornerstone projects in the ecosystem:
    opam: source-based package management
    dune: composable build system for OCaml
    merlin: IDE and type-checker integration

    utop: interactive REPL via a CLI

    ppxlib: AST manipulation infrastructure

    opam-publish: release OCaml packages
    Usage Guidelines:
    Are the recommendation for new projects.

    Advanced users can continue to use toolchain directly.

    Avoid duplication of functionality.

    Metadata files are versioned reliably.

    View Slide

  20. The OCaml Platform: lifecycle
    Incubate Active
    Sustain
    Deprecate
    Filling a gap in the tooling ecosystem:
    odoc: cross-referenced documentation
    ocamlformat: mechanical code formatting
    lsp-server: OCaml backend for IDEs

    mdx: automated testing of code fragments

    bun: automated fuzz testing in CI

    dune-release: easier releases to opam
    How to get promoted?
    Establish a migration path and need.

    odoc: replace all major uses of ocamldoc

    ocamlformat: support partial file indentation (ocp-indent)

    lsp-server: a release cycle for time and testing

    dune-release: integrate with opam-publish, avoid duplication

    View Slide

  21. The OCaml Platform: lifecycle
    Incubate Active
    Sustain
    Deprecate
    Stable and being maintained best-effort:
    ocamlfind: compilation unit manager

    ocamlbuild: build system (formerly included with compiler)
    ocp-indent: as-you-type code formatting
    Sustain projects are best-effort
    If your projects use them, then it is safe to continue, but there may
    be better alternatives.

    ocamlfind: generally no need to use directly

    ocamlbuild: recommend a migration to dune, but older projects will
    still build fine. Newer OCaml options may not be exposed.

    ocp-indent: works fine and stable, but steadily supplanted by
    ocamlformat.

    View Slide

  22. The OCaml Platform: lifecycle
    Incubate Active
    Sustain
    Deprecate
    Deprecated projects:
    camlp4: no longer included in compiler

    or used as a dependency in tools


    oasis: no longer supported for modern

    versions of OCaml
    Deprecated projects are only deprecated for the Platform
    In all cases, any community user can step up and take over
    maintainership. However, the guarantees about being released in
    sync with the OCaml compiler no longer hold.

    camlp4: now maintained by ygrek for the community (thanks!)

    oasis: still builds old code (e.g. Mirage libraries) fine, not used in
    new packages.

    View Slide

  23. The OCaml Platform: innovation
    Incubate Active
    Sustain
    Deprecate
    Balance between progress and stability:
    A thought experiment: if an OCaml Platform

    in 2013 had required ocamlbuild, would
    dune exist now?

    So incubated projects with new points of
    view are vital, but they must progress to
    maturity before replacing active ones.

    Improving syntax and metadata is important.
    Dev meeting notes regularly recorded where practical:
    https://github.com/ocaml/opam/wiki

    https://github.com/ocaml/dune/wiki

    https://github.com/ocaml/odoc/wiki
    Maintainer guidelines:
    Active projects need to remain open to contributors.

    No active OCaml Platform project shall be restricted due to
    employer, and shall retain a liberal license.

    View Slide

  24. The OCaml Platform: maintainers
    odoc documenting Jon Ludlam, Leo White incubate
    ocamlformat editing Josh Berdine, Guillaume Petiot incubate
    lsp-server editing Rudi Grinberg incubate
    mdx testing Guillaume Petiot, Nathan Rebours incubate
    bun testing Mindy Preston incubate
    dune-release packaging Nathan Rebours, Thomas Gazagnaire incubate
    opam packaging Raja Boujbel, Louis Gesbert, David Allsopp active
    opam-publish packaging Louis Gesbert active
    dune building Jeremie Dimino, Rudi Grinberg active
    merlin editing Fred Bour, Thomas Refis active
    utop editing ZAN DoYe, Jeremie Dimino active
    ppxlib ast Jeremie Dimino, Nathan Rebours active
    ocamlfind packaging Gerd Stolpmann sustain
    ocamlbuild building Gabriel Scherer sustain
    ocp-indent editing Louis Gesbert sustain
    omp ast Fred Bour, Jeremie Dimino sustain
    camlp4 ast ygrek deprecate
    These maintainers work for a diverse range of organisations (Tarides, Jane Street,
    OCamlPro, OCaml Labs, Inria, Facebook, OCaml Software Foundation) or act as
    individual contributors.

    View Slide

  25. ocaml.org services
    docs.ocaml.org central repository of API docs incubate
    git.ocaml.org private mirror of repositories incubate
    inbox.ocaml.org email list archives active
    ci.ocaml.org continuous integration for opam active
    discuss.ocaml.org forum for threaded discussion active
    discord.ocaml.org server for Discord chat bot active
    www.ocaml.org main website active
    lists.ocaml.org email list management sustain
    opam.ocaml.org opam package lists sustain
    forge.ocaml.org source code hosting deprecate
    Some services will be deprecated over the next 12 months:
    lists.ocaml.org is quite a maintenance burden

    forge.ocaml.org will finally have DNS entries removed.
    Some services will improve over next 12 months:
    ci.ocaml.org will be replaced with an ocurrent.org-based CI, backed
    by a multiarch computer cluster (see Thomas Leonard's talk)

    docs.ocaml.org: odoc is seeing major improvements to support this
    (see Jon Ludlam's talk)
    Thanks to Packet, Scaleway, Amazon and Cambridge University for sponsorship

    View Slide

  26. The OCaml Platform
    Incubate Active
    Sustain
    Deprecate
    What are the next steps?
    • merge opam.ocaml.org with the main
    OCaml website.

    • add a newsfeed to the OCaml website
    for platform announcements (from
    discuss.ocaml.org).

    • regular newsletter about Platform
    developments (like the "multicore
    monthlies").

    • add workflow guides to ocaml.org
    There are still some major missing development gaps, however:
    Windows support, and better turnkey installers.
    Onto the development plans for the next 12 months...

    View Slide

  27. OCaml Platform

    Development Directions for 2020-2021

    View Slide

  28. OCaml Community Metrics
    total number of opam packages

    View Slide

  29. OCaml Community Metrics
    number of unique opam packages

    View Slide

  30. OCaml Community Metrics
    number of package contributors

    View Slide

  31. several opam 2 betas
    to gather feedback
    and improve workflows
    Feb 2017: opam 2~beta1
    Mar 2017: opam2~beta2
    May 2017: opam2~beta3

    June 2017: opam 1.2.0 deprecated
    Aug 2017: opam 2~beta4

    Nov 2017: opam 2~beta5
    Jan 2018: opam 2~rc1
    May 2018: critical build issue in camlp5

    May 2018: opam 2~rc2 (sandboxing)
    July 2018: opam 2~rc3
    Aug 2018: opam 2~rc4

    Sep 2018: opam 2.0.0 stable released!

    Sep 2018: opam-repo using 2.0 format.
    opam 2.0.0 released
    with v1.2-v2.0 migration
    and secure sandboxing
    Jan 2019: opam 2.0.3
    Apr 2019: opam 2.0.4 (fix pin-depends)
    Jul 2019: opam 2.0.5 (improve linting)
    Jan 2020: opam 2.0.6 (dune caching)

    Apr 2020: opam 2.0.7
    refining and bugfixes

    View Slide

  32. several opam 2 betas
    to gather feedback
    and improve workflows
    Feb 2017: opam 2~beta1
    Mar 2017: opam2~beta2
    May 2017: opam2~beta3

    June 2017: opam 1.2.0 deprecated
    Aug 2017: opam 2~beta4

    Nov 2017: opam 2~beta5
    Jan 2018: opam 2~rc1
    May 2018: critical build issue in camlp5

    May 2018: opam 2~rc2 (sandboxing)
    July 2018: opam 2~rc3
    Aug 2018: opam 2~rc4

    Sep 2018: opam 2.0.0 stable released!

    Sep 2018: opam-repo using 2.0 format.
    opam 2.0.0 released
    with v1.2-v2.0 migration
    and secure sandboxing
    Jan 2019: opam 2.0.3
    Apr 2019: opam 2.0.4 (fix pin-depends)
    Jul 2019: opam 2.0.5 (improve linting)
    Jan 2020: opam 2.0.6 (dune caching)

    Apr 2020: opam 2.0.7
    refining and bugfixes
    opam 2.1.0~beta1!

    View Slide

  33. opam 2.1.0 beta1
    opam 2.1.0 contains hundreds of bug fixes, and these major new features:

    - integrated external dependency handling
    - opam install will intelligently check for apt/rpm/brew depends.

    - no more need for an external opam depext

    - opam lock integration
    - generate fixed version files via "opam lock"

    - create reproducible switches from these lock files.

    - opam cli versioning
    - in scripts, just set OPAM_CLI=2.1 to guarantee 2.1 behaviour, even in future
    clients.

    - switch invariants
    - richer version constraints to specify base packages in a switch, such as
    upgrading automatically on OCaml patch releases.

    - significant performance improvements:
    - faster repository loading

    - interleaving downloading and installation

    - more efficient copying during package installation

    - new "0install solver" optimised for finding from-scratch solutions

    View Slide

  34. opam 2.1.0 beta1
    Also incubating new plugins to support more workflows:

    - opam compiler
    - https://github.com/ocaml-opam/opam-compiler

    - manage OCaml compiler installations much more easily

    - opam tools
    - https://github.com/avsm/opam-tools

    - install Platform tools within a local switch automatically

    - opam monorepo
    - https://github.com/ocamllabs/duniverse

    - generate dune-compatible monorepos as opam lock files

    - (the tool formerly known as duniverse, now integrated with opam)
    These are all under development, and examples of how
    we can experiment with new workflows from within the
    familiar opam framework

    View Slide

  35. opam 2.2, aka the "Windows release"
    As soon as opam 2.1.0 is released, the maintainers have decided that
    we need to focus together on end-to-end Windows support.
    This will integrate fdopen's superb Cygwin fork.
    Requires some key client features, which also help other platforms:

    • Native shell integration
    • Do not assume the existence of a unix environment

    • Package parameters
    • Support variable tracking through opam files

    • Helps rationalise compiler variants as well (500+ now).

    • Layered switches
    • allow binaries to be made available across switches.

    • Build environments
    • generalise sandboxing to support more toolchains

    • could potentially use this to build directly in containers too
    More details:

    https://github.com/ocaml/opam/wiki/opam-2.2-slides.pdf

    View Slide

  36. Le Fin
    See you on discuss.ocaml.org
    and clowdr!

    View Slide