Towards "annex", a Fact Based Dependency System (ICFP)

Slide 1

Slide 1 text

towards annɛx @markhibberd

Slide 2

Slide 2 text

“The enemy is the gramophone mind, whether or not one agrees with the record that is being played at the moment.” George Orwell - The Freedom of the Press

Slide 3

Slide 3 text

one Motivation

Slide 4

Slide 4 text

one Or, we are doing it ALL wrong

Slide 5

Slide 5 text

2.1.3

Slide 6

Slide 6 text

Surely This is a Joke

Slide 7

Slide 7 text

TIME IS A THING

Slide 8

Slide 8 text

commit

Slide 9

Slide 9 text

Slide 10

Slide 10 text

publish

Slide 11

Slide 11 text

platform test

Slide 12

Slide 12 text

production

Slide 13

Slide 13 text

performance

Slide 14

Slide 14 text

cve

Slide 15

Slide 15 text

LANGUAGE WARS ARE PASSE

Slide 16

Slide 16 text

CLOSED WORLD ASSUMPTIONS

Slide 17

Slide 17 text

Dependencies Cost Too Much

Slide 18

Slide 18 text

two Concepts

Slide 19

Slide 19 text

eminence boxer napoleon snowball

Slide 20

Slide 20 text

napoleon boxer snowball eminence wai base

Slide 21

Slide 21 text

napoleon boxer snowball eminence wai base postgres

Slide 22

Slide 22 text

napoleon boxer snowball eminence wai base postgres libpq OS

Slide 23

Slide 23 text

napoleon boxer snowball eminence wai base postgres libpq OS CLANG

Slide 24

Slide 24 text

annex is a fact store

Slide 25

Slide 25 text

START WITH SOMETHING WE HAVE (OR CAN GET)

Slide 26

Slide 26 text

boxer a family

Slide 27

Slide 27 text

family/123-abc-456 a family

Slide 28

Slide 28 text

boxer 1.2.1 an atom

Slide 29

Slide 29 text

atom/123-abc-456 an atom

Slide 30

Slide 30 text

a fact fact/123-abc-457: commit: bd2f074…02

Slide 31

Slide 31 text

fact/123-abc-458: api-signature: […] a fact

Slide 32

Slide 32 text

fact/123-abc-459: feature: it-works a fact

Slide 33

Slide 33 text

We ascribe FACTS to ATOMS

Slide 34

Slide 34 text

atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459

Slide 35

Slide 35 text

boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works

Slide 36

Slide 36 text

The view of FACTS against ATOMS at a point in TIME is a WORLD

Slide 37

Slide 37 text

boxer facts/… snowball facts/… napoleon facts/… a single world

Slide 38

Slide 38 text

Worlds Change Over Time

Slide 39

Slide 39 text

@v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works

Slide 40

Slide 40 text

fact/123-abc-460: tested: FreeBSD-9.1

Slide 41

Slide 41 text

@v1 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works @v2 boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works tested: FreeBSD-9.1

Slide 42

Slide 42 text

Usability #1 Design desisions should be framed in terms of predictability and repeatability

Slide 43

Slide 43 text

annex is a data store

Slide 44

Slide 44 text

Because we believe in interacting with an open world doesn’t mean we have to trust it.

Slide 45

Slide 45 text

atom/123-abc-456 fact/123-abc-457 fact/123-abc-458 fact/123-abc-459 artifact/123-abc-459

Slide 46

Slide 46 text

boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works artifact: tag: bin/boxer flags: […] address: e2f1…bc74

Slide 47

Slide 47 text

points to annex/storage/e2f1…bc74/data /info boxer 1.2.1 artifact: tag: bin/boxer flags: […] address: e2f1…bc74

Slide 48

Slide 48 text

annex/storage/e2f1…bc74/data /info local/storage/e2f1…bc74/data /info free predictable caching

Slide 49

Slide 49 text

Usability #2 Never download something already on a users machine

Slide 50

Slide 50 text

annex is a language

Slide 51

Slide 51 text

:boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 !

Slide 52

Slide 52 text

Usability #3 Precision is important, users should only have to specify what is important to them

Slide 53

Slide 53 text

Usability #4 Flexibility must never come at the cost of determinism

Slide 54

Slide 54 text

First Class Notion of Time

Slide 55

Slide 55 text

:boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 ! +annex.example.com@v12345

Slide 56

Slide 56 text

:boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 +annex.internal.com@v123

Slide 57

Slide 57 text

Queries Across Time

Slide 58

Slide 58 text

:boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” ! :snowball :is :compatable-with atom/… ! :napoleon :semver >= 1.3 < 1.4 :no-cve +annex.example.com@HEAD

Slide 59

Slide 59 text

:no-cve +annex.example.com@HEAD assumes irrefutable facts

Slide 60

Slide 60 text

annex is a tool

Slide 61

Slide 61 text

annex fetch boxer.ax +server@v123

Slide 62

Slide 62 text

annex fetch +server@v123

Slide 63

Slide 63 text

annex fetch -u +server@v123

Slide 64

Slide 64 text

+server@v123 .stable.ax:

Slide 65

Slide 65 text

annex fetch

Slide 66

Slide 66 text

Usability #5 Don’t generate files that a user wouldn’t write by hand

Slide 67

Slide 67 text

annex fetch +repository@v123 annex fetch +repository@v678 annex fetch +repository@v123 Instant By Design

Slide 68

Slide 68 text

annex atom --create family/1d…3b

Slide 69

Slide 69 text

annex fact atom/12…ef feature red

Slide 70

Slide 70 text

git checkout -b topic/feature git add src/Boxer.hs git commit -m ‘Great change!’ git notes add --ref=annex \ -m “:feature win” HEAD git push origin topic/feature git push origin refs/notes/* annex fact atom/12…ef --git HEAD

Slide 71

Slide 71 text

Usability #6 Leverage tools already in use

Slide 72

Slide 72 text

Usability #7 Don’t be as bad as Git

Slide 73

Slide 73 text

three A Deeper Look

Slide 74

Slide 74 text

! resolution

Slide 75

Slide 75 text

annex.mth.io github.com/ambiata/boxer Resolution

Slide 76

Slide 76 text

annex.mth.io github.com/ambiata/boxer Retrieve Facts Resolution

Slide 77

Slide 77 text

annex.mth.io github.com/ambiata/boxer Retrieve Facts Resolution

Slide 78

Slide 78 text

annex.mth.io github.com/ambiata/boxer Send Query Resolution

Slide 79

Slide 79 text

annex.mth.io github.com/ambiata/boxer Synchronize Artifacts Resolution

Slide 80

Slide 80 text

! development

Slide 81

Slide 81 text

Time-Dependent Resolution

Slide 82

Slide 82 text

eminence boxer napoleon snowball

Slide 83

Slide 83 text

napoleon.ax: :boxer :has :feature ingestion ! :snowball :has :feature timeline .stable.ax: +annex.mth.io@v123

Slide 84

Slide 84 text

[ci] stable [ci] stable annex fetch @HEAD annex fetch

Slide 85

Slide 85 text

No content

Slide 86

Slide 86 text

Usability #8 Design for simulation, notifications and metrics

Slide 87

Slide 87 text

An Open World

Slide 88

Slide 88 text

eminence boxer napoleon snowball

Slide 89

Slide 89 text

eminence boxer napoleon snowball _.js wai

Slide 90

Slide 90 text

napoleon.ax: :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs :semver == 1.*

Slide 91

Slide 91 text

napoleon.ax: :wai :source hackage :semver == 2.1.* :underscore.js :source cdnjs :semver == 1.* :tested-on ie4

Slide 92

Slide 92 text

Usability #9 Start with the premise that you need to interact with less principled systems

Slide 93

Slide 93 text

Source Substitution

Slide 94

Slide 94 text

eminence boxer napoleon snowball

Slide 95

Slide 95 text

eminence boxer napoleon snowball Working On A Feature

Slide 96

Slide 96 text

napoleon.ax: :boxer :has :feature ingestion ! :snowball :has :feature timeline ! ! !

Slide 97

Slide 97 text

eminence boxer napoleon snowball Need A Bug Fix

Slide 98

Slide 98 text

annex fetch --source-substitute \ eminence ../eminence

Slide 99

Slide 99 text

annex fetch --source-substitute \ eminence ../eminence\ --ignore-constraints

Slide 100

Slide 100 text

:boxer :has :feature ingestion ! :snowball :has :feature timeline ! :eminence :has :commit ab34…f3e1 :transitive napoleon.ax:

Slide 101

Slide 101 text

annex fact git/HEAD fix ‘#112’

Slide 102

Slide 102 text

:boxer :has :feature ingestion ! :snowball :has :feature timeline ! :eminence :has :fix #112 :transitive napoleon.ax:

Slide 103

Slide 103 text

eminence boxer napoleon snowball Ship It

Slide 104

Slide 104 text

eminence boxer napoleon snowball Never had to touch intermediates

Slide 105

Slide 105 text

Binary Substitution

Slide 106

Slide 106 text

Requires deduction of output signature BEFORE it is built

Slide 107

Slide 107 text

eminence boxer napoleon snowball Depends On Transitives

Slide 108

Slide 108 text

Nix Style Build The World + Better Language Support Essential

Slide 109

Slide 109 text

! distribution

Slide 110

Slide 110 text

If we assume open world, multiple annex fact stores is reality

Slide 111

Slide 111 text

Handling time is non-trivial in a distributed system

Slide 112

Slide 112 text

Annex (currently) chooses federation over being a truly distributed system

Slide 113

Slide 113 text

+red@v123 Time Axis Is Localized To A Given Store

Slide 114

Slide 114 text

Immutability Gives Us ∞ Read Replicas +red@v123 red green blue

Slide 115

Slide 115 text

Query Controlled Writes red green blue +red@v123 => +red@v124

Slide 116

Slide 116 text

Working on a model of facts that always commutes

Slide 117

Slide 117 text

! trust

Slide 118

Slide 118 text

Authenticated FAMILY and ATOM owners

Slide 119

Slide 119 text

Signed Facts

Slide 120

Slide 120 text

Mediation and Fact Views

Slide 121

Slide 121 text

annex.mth.io github.com/ambiata/boxer Fact Mediation annex.inside.ambiata.com [signed-by:…] [owned-by:…]

Slide 122

Slide 122 text

! solving

Slide 123

Slide 123 text

DPLL / SAT based solution w/ inspiration from OPIUM paper

Slide 124

Slide 124 text

Main challenge is mapping fact model to equation

Slide 125

Slide 125 text

Main challenge in mapping fact model is deducing identity from user specified query

Slide 126

Slide 126 text

Secondary challenge is caching partial solutions and reducing duplicated work where possible

Slide 127

Slide 127 text

Reproducibility of solver (a cover up)

Slide 128

Slide 128 text

:boxer :has :feature ingestion ! :snowball :has :feature timeline ! :annex-resolver :has :version 1 napoleon.ax:

Slide 129

Slide 129 text

four A Look Forward

Slide 130

Slide 130 text

0 25 50 75 100 April May June July Deeper Analytics

Slide 131

Slide 131 text

0 25 50 75 100 April May June July Deeper Analytics arbitrary queries and reporting over atoms

Slide 132

Slide 132 text

0 25 50 75 100 April May June July Deeper Analytics inference of relevant facts for customer issues

Slide 133

Slide 133 text

0 25 50 75 100 April May June July Deeper Analytics predict failure in advance

Slide 134

Slide 134 text

Fixing ALL the COMPILERS

Slide 135

Slide 135 text

Easier extension via deductive rules

Slide 136

Slide 136 text

Commutative fact model and non-linear versioning

Slide 137

Slide 137 text

these ideas STEAL

Slide 138

Slide 138 text

end transmission.

Slide 139

Slide 139 text

towards annɛx @markhibberd

Slide 140

Slide 140 text

Images ! Unmodified, Licences Specified at WikiMedia links" http://commons.wikimedia.org/wiki/File:Clock_Gare_de_Paris-Est.jpg! http://en.wikipedia.org/wiki/File:Bundesarchiv_Bild_101III- Merz-014-12A,_Russland,_Beginn_Unternehmen_Zitadelle,_Panzer.jpg! http://commons.wikimedia.org/wiki/File:French_-_Door_with_Cat_Hole_- _Walters_64164.jpg! ! Unmodified. CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)" https://www.flickr.com/photos/timothymorgan/75288582/! https://www.flickr.com/photos/timothymorgan/75288583/! https://www.flickr.com/photos/timothymorgan/75294154/! https://www.flickr.com/photos/timothymorgan/75593155/! https://www.flickr.com/photos/timothymorgan/75593155/