Towards "annex", a Fact Based Dependency System (ICFP)

Knowledge is not static. Yet when dealing with program artifacts, we choose to seal off what we know at the point in time when we know the least. This is wrong.

Context is important. Yet when defining dependencies on artifacts, instead of directly specify the query we want (and hence embedding its context), we manually translate our request into antiquated notions of meta-data, encoded as a number, embedded in a string. Yes, semantic versioning is wrong.

Reproducibility is essential. Yet most existing dependency systems force a trade off of rigour and reproducibility against flexibility and ease of use. This is not necessary.

Drawing on well understood foundations from datalog and deductive databases, and utilizing functional programming fundamentals, "annex" takes a different view on how to manage artifacts. We should be able to ask: "Give me the latest binary compatible versions of X with no known CVE"; or, "Give me the last stable builds of my dependencies that have been tested in IE 9, Chrome and Firefox"; or in a more general context outside of dependency resolution, queries such as "What platforms has build x of my library been tested on?" provide a useful understanding of the current state of artifacts; and finally, it should be possible to phrase all of these questions with a first class notion of time, for example "Give me the same dependencies when I last asked this query".

This talk will start by walking through the concepts behind "annex", before taking a deeper look at the design and implementation (in Haskell). We will look at how its functional underpinnings give rise to very desirable properties for a cross-language dependency system. These properties include: trivial distribution and caching; guaranteed reproducibility with minimal context; predictable performance; and interestingly, how steadfastly holding to functional programming principles contributes to being able to deliver a humane user experience in the face of complexity.

Transcript

1. towards annɛx @markhibberd
2. “The enemy is the gramophone mind, whether or not one agrees with the record that is being played at the moment.” George Orwell - The Freedom of the Press
3. one Motivation
4. one Or, we are doing it ALL wrong
5. 2.1.3
6. Surely This is a Joke
7. TIME IS A THING
8. commit
9. ci
10. publish
11. platform test
12. production
13. performance
14. cve
15. LANGUAGE WARS ARE PASSE
16. CLOSED WORLD ASSUMPTIONS
17. Dependencies Cost Too Much
18. two Concepts
19. eminence boxer napoleon snowball
20. napoleon boxer snowball eminence wai base
21. napoleon boxer snowball eminence wai base postgres
22. napoleon boxer snowball eminence wai base postgres libpq OS
23. napoleon boxer snowball eminence wai base postgres libpq OS CLANG
24. annex is a fact store
25. START WITH SOMETHING WE HAVE (OR CAN GET)
26. boxer a family
27. family/123-abc-456 a family
28. boxer 1.2.1 an atom
29. atom/123-abc-456 an atom
30. a fact fact/123-abc-457: commit: bd2f074…02
31. fact/123-abc-458: api-signature: […] a fact
32. fact/123-abc-459: feature: it-works a fact
33. We ascribe FACTS to ATOMS
34. atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459
35. boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works
36. The view of FACTS against ATOMS at a point in TIME is a WORLD
37. boxer facts/… snowball facts/… napoleon facts/… a single world
38. Worlds Change Over Time
39. @v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works
40. fact/123-abc-460: tested: FreeBSD-9.1
41. @v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works @v2 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works tested: FreeBSD-9.1
42. Usability #1 Design desisions should be framed in terms of predictability and repeatability
43. annex is a data store
44. Because we believe in interacting with an open world doesn’t mean we have to trust it.
45. atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459 artifact/123-abc-459
46. boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works artifact: tag: bin/boxer flags: […] address: e2f1…bc74
47. points to annex/storage/e2f1…bc74/data /info boxer 1.2.1 artifact: tag: bin/boxer flags: […] address: e2f1…bc74
48. annex/storage/e2f1…bc74/data /info local/storage/e2f1…bc74/data /info free predictable caching
49. Usability #2 Never download something already on a users machine
50. annex is a language
51. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 !
52. Usability #3 Precision is important, users should only have to specify what is important to them
53. Usability #4 Flexibility must never come at the cost of determinism
54. First Class Notion of Time
55. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 ! +annex.example.com@v12345
56. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 +annex.internal.com@v123
57. Queries Across Time
58. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 :no-cve +annex.example.com@HEAD
59. :no-cve +annex.example.com@HEAD assumes irrefutable facts
60. annex is a tool
61. annex fetch boxer.ax +server@v123
62. annex fetch +server@v123
63. annex fetch -u +server@v123
64. +server@v123 .stable.ax:
65. annex fetch
66. Usability #5 Don’t generate files that a user wouldn’t write by hand
67. annex fetch +repository@v123 annex fetch +repository@v678 annex fetch +repository@v123 Instant By Design
68. annex atom --create family/1d…3b
69. annex fact atom/12…ef feature red
70. git checkout -b topic/feature git add src/Boxer.hs git commit -m ‘Great change!’ git notes add --ref=annex \ -m “:feature win” HEAD git push origin topic/feature git push origin refs/notes/* annex fact atom/12…ef --git HEAD
71. Usability #6 Leverage tools already in use
72. Usability #7 Don’t be as bad as Git
73. three A Deeper Look
74. ! resolution
75. annex.mth.io github.com/ambiata/boxer Resolution
76. annex.mth.io github.com/ambiata/boxer Retrieve Facts Resolution
77. annex.mth.io github.com/ambiata/boxer Retrieve Facts Resolution
78. annex.mth.io github.com/ambiata/boxer Send Query Resolution
79. annex.mth.io github.com/ambiata/boxer Synchronize Artifacts Resolution
80. ! development
81. Time-Dependent Resolution
82. eminence boxer napoleon snowball
83. napoleon.ax: :boxer :has :feature ingestion ! :snowball :has :feature timeline .stable.ax: +annex.mth.io@v123
84. [ci] stable [ci] stable annex fetch @HEAD annex fetch
86. Usability #8 Design for simulation, notifications and metrics
87. An Open World
88. eminence boxer napoleon snowball
89. eminence boxer napoleon snowball _.js wai
90. napoleon.ax: :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs :semver == 1.*
91. napoleon.ax: :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs :semver == 1.* :tested-on ie4
92. Usability #9 Start with the premise that you need to interact with less principled systems
93. Source Substitution
94. eminence boxer napoleon snowball
95. eminence boxer napoleon snowball Working On A Feature
96. napoleon.ax: :boxer :has :feature ingestion ! :snowball :has :feature timeline ! ! !
97. eminence boxer napoleon snowball Need A Bug Fix
98. annex fetch --source-substitute \ eminence ../eminence
99. annex fetch --source-substitute \ eminence ../eminence\ --ignore-constraints
100. :boxer :has :feature ingestion ! :snowball :has :feature timeline ! :eminence :has :commit ab34…f3e1 :transitive napoleon.ax:
101. annex fact git/HEAD fix ‘#112’
102. :boxer :has :feature ingestion ! :snowball :has :feature timeline ! :eminence :has :fix #112 :transitive napoleon.ax:
103. eminence boxer napoleon snowball Ship It
104. eminence boxer napoleon snowball Never had to touch intermediates
105. Binary Substitution
106. Requires deduction of output signature BEFORE it is built
107. eminence boxer napoleon snowball Depends On Transitives
108. Nix Style Build The World + Better Language Support Essential
109. ! distribution
110. If we assume open world, multiple annex fact stores is reality
111. Handling time is non-trivial in a distributed system
112. Annex (currently) chooses federation over being a truly distributed system
113. +red@v123 Time Axis Is Localized To A Given Store
114. Immutability Gives Us ∞ Read Replicas +red@v123 red green blue
115. Query Controlled Writes red green blue +red@v123 => +red@v124
116. Working on a model of facts that always commutes
117. ! trust
118. Authenticated FAMILY and ATOM owners
119. Signed Facts
120. Mediation and Fact Views
121. annex.mth.io github.com/ambiata/boxer Fact Mediation annex.inside.ambiata.com [signed-by:…] [owned-by:…]
122. ! solving
123. DPLL / SAT based solution w/ inspiration from OPIUM paper
124. Main challenge is mapping fact model to equation
125. Main challenge in mapping fact model is deducing identity from user specified query
126. Secondary challenge is caching partial solutions and reducing duplicated work where possible
127. Reproducibility of solver (a cover up)
128. :boxer :has :feature ingestion ! :snowball :has :feature timeline ! :annex-resolver :has :version 1 napoleon.ax:
129. four A Look Forward
130. 0 25 50 75 100 April May June July Deeper Analytics
131. 0 25 50 75 100 April May June July Deeper Analytics arbitrary queries and reporting over atoms
132. 0 25 50 75 100 April May June July Deeper Analytics inference of relevant facts for customer issues
133. 0 25 50 75 100 April May June July Deeper Analytics predict failure in advance
134. Fixing ALL the COMPILERS
135. Easier extension via deductive rules
136. Commutative fact model and non-linear versioning
137. these ideas STEAL
138. end transmission.
139. towards annɛx @markhibberd
140. Images ! Unmodified, Licences Specified at WikiMedia links" http://commons.wikimedia.org/wiki/File:Clock_Gare_de_Paris-Est.jpg! http://en.wikipedia.org/wiki/File:Bundesarchiv_Bild_101III- Merz-014-12A,_Russland,_Beginn_Unternehmen_Zitadelle,_Panzer.jpg! http://commons.wikimedia.org/wiki/File:French_-_Door_with_Cat_Hole_- _Walters_64164.jpg! ! Unmodified. CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)" https://www.flickr.com/photos/timothymorgan/75288582/! https://www.flickr.com/photos/timothymorgan/75288583/! https://www.flickr.com/photos/timothymorgan/75294154/! https://www.flickr.com/photos/timothymorgan/75593155/! https://www.flickr.com/photos/timothymorgan/75593155/