Towards "annex", a Fact Based Dependency System

Knowledge is not static. Yet when dealing with program artifacts, we choose to seal off what we know at the point in time when we know the least. This is wrong.

Context is important. Yet when defining dependencies on artifacts, instead of directly specify the query we want (and hence embedding its context), we manually translate our request into antiquated notions of meta-data, encoded as a number, embedded in a string. Yes, semantic versioning is wrong.

Reproducibility is essential. Yet most existing dependency systems force a trade off of rigour and reproducibility against flexibility and ease of use. This is not necessary.

Drawing on well understood foundations from datalog and deductive databases, and utilizing functional programming fundamentals, “annex” takes a different view on how to manage artifacts. We should be able to ask: “Give me the latest binary compatible versions of X with no known CVE”; or, “Give me the last stable builds of my dependencies that have been tested in IE 9, Chrome and Firefox”; or in a more general context outside of dependency resolution, queries such as “What platforms has build x of my library been tested on?” provide a useful understanding of the current state of artifacts; and finally, it should be possible to phrase all of these questions with a first class notion of time, for example “Give me the same dependencies when I last asked this query”.

This talk will start by walking through the concepts behind “annex”, before taking a deeper look at the design and implementation (in Haskell) and a multi-language demonstration. We will look at how its functional underpinnings give rise to very desirable properties for a dependency system. These properties include: trivial distribution and caching; guaranteed reproducibility with minimal context; predictable performance; and interestingly, how steadfastly holding to functional programming principles contributes to being able to deliver a humane user experience in the face of complexity.

Presented at strangeloop, 19th of September 2014 - https://thestrangeloop.com/sessions/towards-annex-a-fact-based-dependency-system

Transcript

1. towards
   annɛx
   @markhibberd
   

   View Slide

2. “The enemy is the gramophone
   mind, whether or not one
   agrees with the record that is
   being played at the moment.”
   George Orwell - The Freedom of the Press
   

   View Slide

3. one
   Motivation
   

   View Slide

4. one
   Or, we are doing it ALL wrong
   

   View Slide

5. 2.0.1
   

   View Slide

6. 2.0.2
   

   View Slide

7. 2.1.0
   

   View Slide

8. 3.0.0
   

   View Slide

9. Surely
   This is a Joke
   

   View Slide

10. LANGUAGE WARS ARE PASSE
    

    View Slide

11. TIME IS A THING
    

    View Slide

12. commit
    

    View Slide

13. ci
    

    View Slide

14. publish
    

    View Slide

15. platform test
    

    View Slide

16. production
    

    View Slide

17. performance
    

    View Slide

18. cve
    

    View Slide

19. CLOSED WORLD ASSUMPTIONS
    

    View Slide

20. Dependencies
    Cost Too Much
    

    View Slide

21. two
    Concepts
    

    View Slide

22. eminence
    boxer
    napoleon
    snowball
    

    View Slide

23. napoleon
    boxer snowball
    eminence wai
    base
    

    View Slide

24. napoleon
    boxer snowball
    eminence wai
    base
    postgres
    

    View Slide

25. napoleon
    boxer snowball
    eminence wai
    base
    postgres
    libpq
    OS
    

    View Slide

26. napoleon
    boxer snowball
    eminence wai
    base
    postgres
    libpq
    OS
    CLANG
    

    View Slide

27. annex is a
    fact store
    

    View Slide

28. START WITH SOMETHING WE
    HAVE (OR CAN GET)
    

    View Slide

29. boxer
    a family
    

    View Slide

30. family/123-abc-456
    a family
    

    View Slide

31. boxer 1.2.1
    an atom
    

    View Slide

32. atom/123-abc-456
    an atom
    

    View Slide

33. a fact
    fact/123-abc-457:
    commit: bd2f074…02
    

    View Slide

34. fact/123-abc-458:
    api-signature: […]
    a fact
    

    View Slide

35. fact/123-abc-459:
    feature: it-works
    a fact
    

    View Slide

36. We ascribe FACTS to ATOMS
    

    View Slide

37. atom/123-abc-456
    fact/123-abc-457
    fact/123-abc-458
    fact/123-abc-459
    

    View Slide

38. boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    

    View Slide

39. The view of FACTS against ATOMS
    at a point in TIME is a WORLD
    

    View Slide

40. boxer
    facts/…
    snowball
    facts/…
    napoleon
    facts/…
    a single world
    

    View Slide

41. Worlds Change Over Time
    

    View Slide

42. @v1
    boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    

    View Slide

43. fact/123-abc-460:
    tested: FreeBSD-9.1
    

    View Slide

44. @v1
    boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    @v2
    boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    tested: FreeBSD-9.1
    

    View Slide

45. Usability #1
    Design desisions should be
    framed in terms of
    predictability and repeatability
    

    View Slide

46. annex is a
    data store
    

    View Slide

47. Because we believe in
    interacting with an open world
    doesn’t mean we have to trust it.
    

    View Slide

48. atom/123-abc-456
    fact/123-abc-457
    fact/123-abc-458
    fact/123-abc-459
    artifact/123-abc-459
    

    View Slide

49. boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    artifact:
    tag: bin/boxer
    flags: […]
    address: e2f1…bc74
    

    View Slide

50. points to
    annex/storage/e2f1…bc74/data
    /info
    boxer 1.2.1
    artifact:
    tag: bin/boxer
    flags: […]
    address: e2f1…bc74
    

    View Slide

51. annex/storage/e2f1…bc74/data
    /info
    local/storage/e2f1…bc74/data
    /info
    free predictable caching
    

    View Slide

52. Usability #2
    Never download something
    already on a users machine
    

    View Slide

53. annex is a
    language
    

    View Slide

54. :boxer
    :has :feature “multi-part-put”
    :has :commit “abcd-1345”
    :has :branch “master”
    !
    :snowball
    :is :compatable-with atom/…
    !
    :napoleon
    :semver >= 1.3 < 1.4
    !
    

    View Slide

55. First Class Notion of Time
    

    View Slide

56. :boxer
    :has :feature “multi-part-put”
    :has :commit “abcd-1345”
    :has :branch “master”
    !
    :snowball
    :is :compatable-with atom/…
    !
    :napoleon
    :semver >= 1.3 < 1.4
    !
    +annex.example.com@v12345
    

    View Slide

57. :boxer
    :has :feature “multi-part-put”
    :has :commit “abcd-1345”
    :has :branch “master”
    !
    :snowball
    :is :compatable-with atom/…
    !
    :napoleon
    :semver >= 1.3 < 1.4
    +annex.internal.com@v123
    

    View Slide

58. :boxer
    :has :feature “multi-part-put”
    :has :commit “abcd-1345”
    :has :branch “master”
    !
    :snowball
    :is :compatable-with atom/…
    !
    :napoleon
    :semver >= 1.3 < 1.4
    :no-cve +annex.example.com@HEAD
    

    View Slide

59. :no-cve +annex.example.com@HEAD
    assumes irrefutable facts
    

    View Slide

60. Usability #3
    Precision is important, users
    should only have to specify
    what is important to them
    

    View Slide

61. Usability #4
    Flexibility must never come
    at the cost of determinism
    

    View Slide

62. annex is a
    tool
    

    View Slide

63. annex fetch boxer.ax +server@v123
    

    View Slide

64. annex fetch +server@v123
    

    View Slide

65. annex fetch -u +server@v123
    

    View Slide

66. +server@v123
    .stable.ax:
    

    View Slide

67. annex fetch
    

    View Slide

68. Usability #5
    Don’t generate files that a
    user wouldn’t write by hand
    

    View Slide

69. annex fetch +repository@v123
    annex fetch +repository@v678
    annex fetch +repository@v123
    Instant By Design
    

    View Slide

70. annex atom --create family/1d…3b
    

    View Slide

71. annex fact atom/12…ef feature red
    

    View Slide

72. git checkout -b topic/feature
    git add src/Boxer.hs
    git commit -m ‘Great change!
    fixes #12
    annex:
    :feature winning’
    git push origin topic/feature
    annex fact atom/12…ef --git HEAD
    

    View Slide

73. git notes add --ref=annex \
    -m “:feature again” HEAD
    git push origin refs/notes/*
    annex fact atom/12…ef --git HEAD
    

    View Slide

74. Usability #6
    Leverage tools already in use
    

    View Slide

75. Usability #7
    Don’t be as bad as Git
    

    View Slide

76. !
    scalaVersion := 2.11
    !
    annexDependencies := List(
    atom(“ivory”)
    .has(“feature”, “puts”)
    )
    build.sbt:
    

    View Slide

77. !
    name: napoleon
    depends-on: eminence
    :feature fix-#12
    napoleon.cabal.annex:
    

    View Slide

78. three
    A Deeper Look
    

    View Slide

79. !
    resolution
    

    View Slide

80. annex.mth.io
    github.com/ambiata/boxer
    Resolution
    

    View Slide

81. annex.mth.io
    github.com/ambiata/boxer
    Retrieve Facts
    Resolution
    

    View Slide

82. annex.mth.io
    github.com/ambiata/boxer
    Retrieve Facts
    Resolution
    

    View Slide

83. annex.mth.io
    github.com/ambiata/boxer
    Send Query
    Resolution
    

    View Slide

84. annex.mth.io
    github.com/ambiata/boxer
    Synchronize Artifacts
    Resolution
    

    View Slide

85. !
    development
    

    View Slide

86. Time-Dependent Resolution
    

    View Slide

87. eminence
    boxer
    napoleon
    snowball
    

    View Slide

88. napoleon.ax:
    :boxer
    :has :feature ingestion
    !
    :snowball
    :has :feature timeline
    .stable.ax:
    +annex.mth.io@v123
    

    View Slide

89. [ci] stable
    [ci] edge
    annex fetch @HEAD
    annex fetch
    

    View Slide

90. View Slide

91. Usability #8
    Design for simulation,
    notifications and metrics
    

    View Slide

92. An Open World
    

    View Slide

93. eminence
    boxer
    napoleon
    snowball
    

    View Slide

94. eminence
    boxer
    napoleon
    snowball
    _.js
    wai
    

    View Slide

95. napoleon.ax:
    :wai
    :source hackage
    :semver == 2.1.*
    :underscore.js
    :source cdnjs
    :semver == 1.*
    

    View Slide

96. napoleon.ax:
    :wai
    :source hackage
    :semver == 2.1.*
    :underscore.js
    :source cdnjs
    :semver == 1.*
    :tested-on ie4
    

    View Slide

97. Usability #9
    Start with the premise that
    you need to interact with
    less principled systems
    

    View Slide

98. Source Substitution
    

    View Slide

99. eminence
    boxer
    napoleon
    snowball
    

    View Slide

100. eminence
     boxer
     napoleon
     snowball
     Working On
     A Feature
     

     View Slide

101. napoleon.ax:
     :boxer
     :has :feature ingestion
     !
     :snowball
     :has :feature timeline
     !
     !
     !
     

     View Slide

102. eminence
     boxer
     napoleon
     snowball
     Need A
     Bug Fix
     

     View Slide

103. annex fetch --source-substitute \
     eminence ../eminence
     

     View Slide

104. annex fetch --source-substitute \
     eminence ../eminence\
     --ignore-constraints
     

     View Slide

105. :boxer
     :has :feature ingestion
     !
     :snowball
     :has :feature timeline
     !
     :eminence
     :has :commit ab34…f3e1
     :transitive
     napoleon.ax:
     

     View Slide

106. annex fact git/HEAD fix ‘#112’
     

     View Slide

107. :boxer
     :has :feature ingestion
     !
     :snowball
     :has :feature timeline
     !
     :eminence
     :has :fix #112
     :transitive
     napoleon.ax:
     

     View Slide

108. eminence
     boxer
     napoleon
     snowball
     Ship It
     

     View Slide

109. eminence
     boxer
     napoleon
     snowball
     Never had to touch
     intermediates
     

     View Slide

110. Binary Substitution
     

     View Slide

111. Requires deduction of output
     signature BEFORE it is built
     

     View Slide

112. eminence
     boxer
     napoleon
     snowball
     Depends On
     Transitives
     

     View Slide

113. Nix Style Build The World +
     Better Language Support
     Essential
     

     View Slide

114. !
     distribution
     

     View Slide

115. If we assume open world,
     multiple annex fact stores
     is reality
     

     View Slide

116. Handling time is non-trivial
     in a distributed system
     

     View Slide

117. Annex (currently) chooses
     federation over being a truly
     distributed system
     

     View Slide

118. +red@v123
     Time Axis Is
     Localized To
     A Given Store
     

     View Slide

119. Immutability Gives Us ∞ Read Replicas
     +red@v123
     red green blue
     

     View Slide

120. Query Controlled Writes
     red green blue
     +red@v123 => +red@v124
     

     View Slide

121. Working on a model of facts that
     always commutes
     

     View Slide

122. !
     trust
     

     View Slide

123. Authenticated
     FAMILY and ATOM owners
     

     View Slide

124. Signed Facts
     

     View Slide

125. Mediation and Fact Views
     

     View Slide

126. annex.mth.io
     github.com/ambiata/boxer
     Fact Mediation
     annex.inside.ambiata.com
     [signed-by:…]
     [owned-by:…]
     

     View Slide

127. !
     solving
     

     View Slide

128. DPLL / SAT based solution w/
     inspiration from OPIUM paper
     

     View Slide

129. Main challenge is mapping fact
     model to equation
     

     View Slide

130. Main challenge in mapping fact
     model is deducing identity from
     user specified query
     

     View Slide

131. Secondary challenge is caching
     partial solutions and reducing
     duplicated work where possible
     

     View Slide

132. Reproducibility of solver
     (a cover up)
     

     View Slide

133. :boxer
     :has :feature ingestion
     !
     :snowball
     :has :feature timeline
     !
     :annex-resolver
     :has :version 1
     napoleon.ax:
     

     View Slide

134. four
     A Look Forward
     

     View Slide

135. 0
     25
     50
     75
     100
     April May June July
     Deeper Analytics
     

     View Slide

136. 0
     25
     50
     75
     100
     April May June July
     Deeper Analytics
     arbitrary queries
     and reporting
     over atoms
     

     View Slide

137. 0
     25
     50
     75
     100
     April May June July
     Deeper Analytics
     inference of
     relevant facts
     for customer issues
     

     View Slide

138. 0
     25
     50
     75
     100
     April May June July
     Deeper Analytics
     predict failure
     in advance
     

     View Slide

139. Fixing ALL the COMPILERS
     

     View Slide

140. Easier extension via
     deductive rules
     

     View Slide

141. Commutative fact model and
     non-linear versioning
     

     View Slide

142. Better Support for
     Operational / Runtime
     Dependencies
     

     View Slide

143. these ideas
     STEAL
     

     View Slide

144. end transmission.
     

     View Slide

145. towards
     annɛx
     @markhibberd
     

     View Slide

146. Images
     !
     Unmodified, Licences Specified at WikiMedia links"
     http://commons.wikimedia.org/wiki/File:Clock_Gare_de_Paris-Est.jpg!
     http://en.wikipedia.org/wiki/File:Bundesarchiv_Bild_101III-
     Merz-014-12A,_Russland,_Beginn_Unternehmen_Zitadelle,_Panzer.jpg!
     http://commons.wikimedia.org/wiki/File:French_-_Door_with_Cat_Hole_-
     _Walters_64164.jpg!
     !
     Unmodified. CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)"
     https://www.flickr.com/photos/timothymorgan/75288582/!
     https://www.flickr.com/photos/timothymorgan/75288583/!
     https://www.flickr.com/photos/timothymorgan/75294154/!
     https://www.flickr.com/photos/timothymorgan/75593155/!
     https://www.flickr.com/photos/timothymorgan/75593155/
     

     View Slide