JS-GIT - Speaker Deck

Slide 1

Slide 1 text

@isntitvacant NODEPDX 2013 Git JS Thursday, May 16, 13

Slide 2

Slide 2 text

@isntitvacant NODEPDX 2013 Thursday, May 16, 13

Slide 3

Slide 3 text

@isntitvacant NODEPDX 2013 HI.my name is chris. I work at . (it’s fun) (As a JS engineer) (We're hiring) Thursday, May 16, 13

Slide 4

Slide 4 text

@isntitvacant NODEPDX 2013 wHY GIT? Thursday, May 16, 13

Slide 5

Slide 5 text

@isntitvacant NODEPDX 2013 Made a GIT/HG site in 2010 with a friend “repocracy” very nearly immediately trolled wHY? Thursday, May 16, 13

Slide 6

Slide 6 text

@isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

Slide 7

Slide 7 text

@isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

Slide 8

Slide 8 text

@isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

Slide 9

Slide 9 text

@isntitvacant NODEPDX 2013 wHY? wanted to limit repo size before cloning hence, “tempisfugit” github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

Slide 10

Slide 10 text

@isntitvacant NODEPDX 2013 wHY? tempisfugit was awful, just awful no zlib in node yet heavily tied to fs api monolithic And, of course... github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

Slide 11

Slide 11 text

@isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? Thursday, May 16, 13

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

@isntitvacant NODEPDX 2013 wHY? 2-3 months ago js-git was funded wanted to “back with code” also to apply experience working on voxel.js Thursday, May 16, 13

Slide 15

Slide 15 text

@isntitvacant NODEPDX 2013 WHAT is git? Thursday, May 16, 13

Slide 16

Slide 16 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS Thursday, May 16, 13

Slide 17

Slide 17 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TAGS Thursday, May 16, 13

Slide 18

Slide 18 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS COMMITS TAGS Thursday, May 16, 13

Slide 19

Slide 19 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS Thursday, May 16, 13

Slide 20

Slide 20 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS BLOBS Thursday, May 16, 13

Slide 21

Slide 21 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS BLOBS object 171d82736413d1f781a16777bb67c270de9cc084 type commit tag v0.0.1 tagger Chris Dickinson \ \ Sun Mar 24 00:10:34 2013 -0700 0.0.1 Thursday, May 16, 13

Slide 22

Slide 22 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS BLOBS tree 27bae442f94a74dd7b070d8a37aa666665e6e123 parent dfa6cf3ebb37c53825035ca67482d3fc4c17e029 author Chris Dickinson \ \ 1364109034 -0700 committer Chris Dickinson \ \ 1364109034 -0700 0.0.1 Thursday, May 16, 13

Slide 23

Slide 23 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS BLOBS 100644 blob f9fe4b03cd774c735d4eda711fbadc199c1f77af README.md 100644 blob 3ec1605cfba1dee1a31c76ccbefcd84b7a285d28 examples.js 100644 blob b5e785955ecd0329f6b771f32a7548f1a65e210a index.js 100644 blob 04e7de77ff4351c08fd37aa2079fd6f23a19e9c0 package.json Thursday, May 16, 13

Slide 24

Slide 24 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS BLOBS { "name": "git-fs-repo", "version": "0.0.1", "description": "filesystem backed git repository", "main": "index.js", "scripts": { "test": "node test.js" }, ... Thursday, May 16, 13

Slide 25

Slide 25 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES Thursday, May 16, 13

Slide 26

Slide 26 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES LOOSE Thursday, May 16, 13

Slide 27

Slide 27 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE Thursday, May 16, 13

Slide 28

Slide 28 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE .git/objects/{2 hex}/{38 hex} “commiT”, “BLOB”, “TAG”, “TREE” literal DECIMAL UNCOMPRESSED SIZE DEFLATED OBJECT DATA Thursday, May 16, 13

Slide 29

Slide 29 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE .git/objects/pack/{pack-hash}.idx .git/objects/pack/{pack-hash}.pack IDX =”index” random access for git objects in pack this is where all the deltas live more on packfiles later! Thursday, May 16, 13

Slide 30

Slide 30 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES var log = require('git-walk-refs') var repo = require('git-fs-repo') repo('path/to/.git', function(err, git) { var hashes = git.refs().map(function(ref) { return ref.hash }) log(git.find, hashes) .on('data', console.log) }) Thursday, May 16, 13

Slide 31

Slide 31 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT Thursday, May 16, 13

Slide 32

Slide 32 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES Thursday, May 16, 13

Slide 33

Slide 33 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 4 bytes: “PACK” 4 bytes: version 4 bytes: # of objs HEADER Thursday, May 16, 13

Slide 34

Slide 34 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 1 bit: continuation 3 bits: obj type 4 + N bits: obj size DEFLATED OBJ DATA PACKFILES Object Thursday, May 16, 13

Slide 35

Slide 35 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 20 bytes: checksum checksum Thursday, May 16, 13

Slide 36

Slide 36 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 4 valid types + 2 delta types “offset” and “reference” delta deltas are xdelta format reference has a hash, offset is an offset reference to a previous obj Thursday, May 16, 13

Slide 37

Slide 37 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... deltas may reference other deltas the packing heuristics are neat ... though totally optional Thursday, May 16, 13

Slide 38

Slide 38 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var fs = require('fs') var list = require('git-list-pack') var objectify = require('git-objectify-pack') var objects = {} var find = function(hash, ready) { return ready(null, objects[hash]) } fs.createReadStream('.git/objects/XXX.pack') .pipe(list()) .pipe(objectify(find)) .on('data', function(obj) { objects[obj.hash] = obj }) Thursday, May 16, 13

Slide 39

Slide 39 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn = net.connect({host: 'github.com', port: 9418}) var fetch = require('git-fetch-pack') var transport = require('git-transport-protocol') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/(master)/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(yourFavoriteKVStore()) Thursday, May 16, 13

Slide 40

Slide 40 text

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn = shoe.connect('/git') var fetch = require('git-fetch-pack') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/master/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(levelJS()) Thursday, May 16, 13

Slide 41

Slide 41 text

@isntitvacant NODEPDX 2013 Lessons. Thursday, May 16, 13

Slide 42

Slide 42 text

@isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

Slide 43

Slide 43 text

@isntitvacant NODEPDX 2013 write modules lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

Slide 44

Slide 44 text

@isntitvacant NODEPDX 2013 write modules (LOTS OF small) lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

Slide 45

Slide 45 text

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

Slide 46

Slide 46 text

@isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

Slide 47

Slide 47 text

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

Slide 48

Slide 48 text

@isntitvacant NODEPDX 2013 lessons. easier to test Thursday, May 16, 13

Slide 49

Slide 49 text

@isntitvacant NODEPDX 2013 lessons. easier to test easier to write Thursday, May 16, 13

Slide 50

Slide 50 text

@isntitvacant NODEPDX 2013 lessons. easier to test easier to write easier to document Thursday, May 16, 13

Slide 51

Slide 51 text

@isntitvacant NODEPDX 2013 lessons. easier to test easier to write easier to document easier to change later Thursday, May 16, 13

Slide 52

Slide 52 text

@isntitvacant NODEPDX 2013 lessons. USE Thursday, May 16, 13

Slide 53

Slide 53 text

@isntitvacant NODEPDX 2013 lessons. USE github.com/substack/node-browserify Thursday, May 16, 13

Slide 54

Slide 54 text

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

Slide 55

Slide 55 text

@isntitvacant NODEPDX 2013 great build tool (with standalone) lessons. Thursday, May 16, 13

Slide 56

Slide 56 text

@isntitvacant NODEPDX 2013 great build tool (with standalone) great for shimming lessons. Thursday, May 16, 13

Slide 57

Slide 57 text

@isntitvacant NODEPDX 2013 great build tool (with standalone) great for shimming great tooling (source maps, beefy, testling ci) lessons. Thursday, May 16, 13

Slide 58

Slide 58 text

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

Slide 59

Slide 59 text

@isntitvacant NODEPDX 2013 lessons. put apis in terms of Thursday, May 16, 13

Slide 60

Slide 60 text

@isntitvacant NODEPDX 2013 lessons. put apis in terms of streams Thursday, May 16, 13

Slide 61

Slide 61 text

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

Slide 62

Slide 62 text

@isntitvacant NODEPDX 2013 lessons. pluggable api Thursday, May 16, 13

Slide 63

Slide 63 text

@isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire thing working to test Thursday, May 16, 13

Slide 64

Slide 64 text

@isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire thing working to test don’t need all of the data at once to process Thursday, May 16, 13

Slide 65

Slide 65 text

@isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire thing working to test don’t need all of the data at once to process use through! Thursday, May 16, 13

Slide 66

Slide 66 text

@isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire thing working to test don’t need all of the data at once to process use through! (and min-streams underneath) Thursday, May 16, 13

Slide 67

Slide 67 text

@isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

Slide 68

Slide 68 text

@isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

Slide 69

Slide 69 text

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson Thursday, May 16, 13

Slide 70

Slide 70 text

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant Thursday, May 16, 13

Slide 71

Slide 71 text

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git / freenode Thursday, May 16, 13

Slide 72

Slide 72 text

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git / freenode and thanks to creationix, dominictarr, danlucraft, maksimlin, and others! Thursday, May 16, 13

Slide 73

Slide 73 text

@isntitvacant NODEPDX 2013 ISSUES. Thursday, May 16, 13

Slide 74

Slide 74 text

@isntitvacant NODEPDX 2013 lessons. ZLIB Thursday, May 16, 13

Slide 75

Slide 75 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

Slide 76

Slide 76 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s Thursday, May 16, 13

Slide 77

Slide 77 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s BUT you can’t get the unused bytes from inflate Thursday, May 16, 13

Slide 78

Slide 78 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

Slide 79

Slide 79 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate Thursday, May 16, 13

Slide 80

Slide 80 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me sad Thursday, May 16, 13

Slide 81

Slide 81 text

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me sad but also happy! was fun. github.com/CHRISDICKINSON/inflate Thursday, May 16, 13

Slide 82

Slide 82 text

@isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16, 13

Slide 83

Slide 83 text

@isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16, 13

Slide 84

Slide 84 text

@isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere bu ff ers Thursday, May 16, 13

Slide 85

Slide 85 text

@isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere BUT typed arrays are better in the browser github.com/CHRISDICKINSON/bops bu ff ers Thursday, May 16, 13

Slide 86

Slide 86 text

@isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16, 13

Slide 87

Slide 87 text

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops bu ff ers Thursday, May 16, 13

Slide 88

Slide 88 text

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module bu ff ers Thursday, May 16, 13

Slide 89

Slide 89 text

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module “bops” - buffer ops bu ff ers Thursday, May 16, 13

Slide 90

Slide 90 text

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module “bops” - buffer ops use typed arrays in browser, buffers elsewhere bu ff ers Thursday, May 16, 13

Slide 91

Slide 91 text

@isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

Slide 92

Slide 92 text

@isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

Slide 93

Slide 93 text

@isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive is difficult Thursday, May 16, 13

Slide 94

Slide 94 text

@isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive is difficult requires making all modules dep on it Thursday, May 16, 13

Slide 95

Slide 95 text

@isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive is difficult requires making all modules dep on it and bumping those deps Thursday, May 16, 13

Slide 96

Slide 96 text

@isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive is difficult requires making all modules dep on it and bumping those deps voxel.js ran into this with three.js Thursday, May 16, 13

Slide 97

Slide 97 text

@isntitvacant NODEPDX 2013 ISSUES. small modules Thursday, May 16, 13

Slide 98

Slide 98 text

@isntitvacant NODEPDX 2013 ISSUES. not insurmountable, though small modules Thursday, May 16, 13