Towards "annex", a Fact Based Dependency System (ICFP)

42d9867a0fee0fa6de6534e9df0f1e9b?s=47 Mark Hibberd
September 06, 2014

Towards "annex", a Fact Based Dependency System (ICFP)

Knowledge is not static. Yet when dealing with program artifacts, we choose to seal off what we know at the point in time when we know the least. This is wrong.

Context is important. Yet when defining dependencies on artifacts, instead of directly specify the query we want (and hence embedding its context), we manually translate our request into antiquated notions of meta-data, encoded as a number, embedded in a string. Yes, semantic versioning is wrong.

Reproducibility is essential. Yet most existing dependency systems force a trade off of rigour and reproducibility against flexibility and ease of use. This is not necessary.

Drawing on well understood foundations from datalog and deductive databases, and utilizing functional programming fundamentals, "annex" takes a different view on how to manage artifacts. We should be able to ask: "Give me the latest binary compatible versions of X with no known CVE"; or, "Give me the last stable builds of my dependencies that have been tested in IE 9, Chrome and Firefox"; or in a more general context outside of dependency resolution, queries such as "What platforms has build x of my library been tested on?" provide a useful understanding of the current state of artifacts; and finally, it should be possible to phrase all of these questions with a first class notion of time, for example "Give me the same dependencies when I last asked this query".

This talk will start by walking through the concepts behind "annex", before taking a deeper look at the design and implementation (in Haskell). We will look at how its functional underpinnings give rise to very desirable properties for a cross-language dependency system. These properties include: trivial distribution and caching; guaranteed reproducibility with minimal context; predictable performance; and interestingly, how steadfastly holding to functional programming principles contributes to being able to deliver a humane user experience in the face of complexity.


Mark Hibberd

September 06, 2014


  1. towards annɛx @markhibberd

  2. “The enemy is the gramophone mind, whether or not one

    agrees with the record that is being played at the moment.” George Orwell - The Freedom of the Press
  3. one Motivation

  4. one Or, we are doing it ALL wrong

  5. 2.1.3

  6. Surely This is a Joke


  8. commit

  9. ci

  10. publish

  11. platform test

  12. production

  13. performance

  14. cve



  17. Dependencies Cost Too Much

  18. two Concepts

  19. eminence boxer napoleon snowball

  20. napoleon boxer snowball eminence wai base

  21. napoleon boxer snowball eminence wai base postgres

  22. napoleon boxer snowball eminence wai base postgres libpq OS

  23. napoleon boxer snowball eminence wai base postgres libpq OS CLANG

  24. annex is a fact store


  26. boxer a family

  27. family/123-abc-456 a family

  28. boxer 1.2.1 an atom

  29. atom/123-abc-456 an atom

  30. a fact fact/123-abc-457: commit: bd2f074…02

  31. fact/123-abc-458: api-signature: […] a fact

  32. fact/123-abc-459: feature: it-works a fact

  33. We ascribe FACTS to ATOMS

  34. atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459

  35. boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works

  36. The view of FACTS against ATOMS at a point in

    TIME is a WORLD
  37. boxer facts/… snowball facts/… napoleon facts/… a single world

  38. Worlds Change Over Time

  39. @v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works

  40. fact/123-abc-460: tested: FreeBSD-9.1

  41. @v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works @v2

    boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works tested: FreeBSD-9.1
  42. Usability #1 Design desisions should be framed in terms of

    predictability and repeatability
  43. annex is a data store

  44. Because we believe in interacting with an open world doesn’t

    mean we have to trust it.
  45. atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459 artifact/123-abc-459

  46. boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works artifact: tag:

    bin/boxer flags: […] address: e2f1…bc74
  47. points to annex/storage/e2f1…bc74/data /info boxer 1.2.1 artifact: tag: bin/boxer flags:

    […] address: e2f1…bc74
  48. annex/storage/e2f1…bc74/data /info local/storage/e2f1…bc74/data /info free predictable caching

  49. Usability #2 Never download something already on a users machine

  50. annex is a language

  51. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master”

    ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 !
  52. Usability #3 Precision is important, users should only have to

    specify what is important to them
  53. Usability #4 Flexibility must never come at the cost of

  54. First Class Notion of Time

  55. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master”

    ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 !
  56. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master”

    ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4
  57. Queries Across Time

  58. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master”

    ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 :no-cve
  59. :no-cve assumes irrefutable facts

  60. annex is a tool

  61. annex fetch +server@v123

  62. annex fetch +server@v123

  63. annex fetch -u +server@v123

  64. +server@v123

  65. annex fetch

  66. Usability #5 Don’t generate files that a user wouldn’t write

    by hand
  67. annex fetch +repository@v123 annex fetch +repository@v678 annex fetch +repository@v123 Instant

    By Design
  68. annex atom --create family/1d…3b

  69. annex fact atom/12…ef feature red

  70. git checkout -b topic/feature git add src/Boxer.hs git commit -m

    ‘Great change!’ git notes add --ref=annex \ -m “:feature win” HEAD git push origin topic/feature git push origin refs/notes/* annex fact atom/12…ef --git HEAD
  71. Usability #6 Leverage tools already in use

  72. Usability #7 Don’t be as bad as Git

  73. three A Deeper Look

  74. ! resolution

  75. Resolution

  76. Retrieve Facts Resolution

  77. Retrieve Facts Resolution

  78. Send Query Resolution

  79. Synchronize Artifacts Resolution

  80. ! development

  81. Time-Dependent Resolution

  82. eminence boxer napoleon snowball

  83. :boxer :has :feature ingestion ! :snowball :has :feature timeline
  84. [ci] stable [ci] stable annex fetch @HEAD annex fetch

  85. None
  86. Usability #8 Design for simulation, notifications and metrics

  87. An Open World

  88. eminence boxer napoleon snowball

  89. eminence boxer napoleon snowball _.js wai

  90. :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs

    :semver == 1.*
  91. :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs

    :semver == 1.* :tested-on ie4
  92. Usability #9 Start with the premise that you need to

    interact with less principled systems
  93. Source Substitution

  94. eminence boxer napoleon snowball

  95. eminence boxer napoleon snowball Working On A Feature

  96. :boxer :has :feature ingestion ! :snowball :has :feature timeline

    ! ! !
  97. eminence boxer napoleon snowball Need A Bug Fix

  98. annex fetch --source-substitute \ eminence ../eminence

  99. annex fetch --source-substitute \ eminence ../eminence\ --ignore-constraints

  100. :boxer :has :feature ingestion ! :snowball :has :feature timeline !

    :eminence :has :commit ab34…f3e1 :transitive
  101. annex fact git/HEAD fix ‘#112’

  102. :boxer :has :feature ingestion ! :snowball :has :feature timeline !

    :eminence :has :fix #112 :transitive
  103. eminence boxer napoleon snowball Ship It

  104. eminence boxer napoleon snowball Never had to touch intermediates

  105. Binary Substitution

  106. Requires deduction of output signature BEFORE it is built

  107. eminence boxer napoleon snowball Depends On Transitives

  108. Nix Style Build The World + Better Language Support Essential

  109. ! distribution

  110. If we assume open world, multiple annex fact stores is

  111. Handling time is non-trivial in a distributed system

  112. Annex (currently) chooses federation over being a truly distributed system

  113. +red@v123 Time Axis Is Localized To A Given Store

  114. Immutability Gives Us ∞ Read Replicas +red@v123 red green blue

  115. Query Controlled Writes red green blue +red@v123 => +red@v124

  116. Working on a model of facts that always commutes

  117. ! trust

  118. Authenticated FAMILY and ATOM owners

  119. Signed Facts

  120. Mediation and Fact Views

  121. Fact Mediation [signed-by:…] [owned-by:…]

  122. ! solving

  123. DPLL / SAT based solution w/ inspiration from OPIUM paper

  124. Main challenge is mapping fact model to equation

  125. Main challenge in mapping fact model is deducing identity from

    user specified query
  126. Secondary challenge is caching partial solutions and reducing duplicated work

    where possible
  127. Reproducibility of solver (a cover up)

  128. :boxer :has :feature ingestion ! :snowball :has :feature timeline !

    :annex-resolver :has :version 1
  129. four A Look Forward

  130. 0 25 50 75 100 April May June July Deeper

  131. 0 25 50 75 100 April May June July Deeper

    Analytics arbitrary queries and reporting over atoms
  132. 0 25 50 75 100 April May June July Deeper

    Analytics inference of relevant facts for customer issues
  133. 0 25 50 75 100 April May June July Deeper

    Analytics predict failure in advance
  134. Fixing ALL the COMPILERS

  135. Easier extension via deductive rules

  136. Commutative fact model and non-linear versioning

  137. these ideas STEAL

  138. end transmission.

  139. towards annɛx @markhibberd

  140. Images ! Unmodified, Licences Specified at WikiMedia links"!

    Merz-014-12A,_Russland,_Beginn_Unternehmen_Zitadelle,_Panzer.jpg! _Walters_64164.jpg! ! Unmodified. CC BY 2.0 ("!!!!