JS-GIT

Transcript

1. @isntitvacant NODEPDX 2013 Git JS Thursday, May 16, 13

2. @isntitvacant NODEPDX 2013 Thursday, May 16, 13

3. @isntitvacant NODEPDX 2013 HI.my name is chris. I work at

   . (it’s fun) (As a JS engineer) (We're hiring) Thursday, May 16, 13

4. @isntitvacant NODEPDX 2013 wHY GIT? Thursday, May 16, 13

5. @isntitvacant NODEPDX 2013 Made a GIT/HG site in 2010 with

   a friend “repocracy” very nearly immediately trolled wHY? Thursday, May 16, 13

6. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

7. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

8. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

9. @isntitvacant NODEPDX 2013 wHY? wanted to limit repo size before

   cloning hence, “tempisfugit” github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

10. @isntitvacant NODEPDX 2013 wHY? tempisfugit was awful, just awful no

    zlib in node yet heavily tied to fs api monolithic And, of course... github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

11. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? Thursday, May 16, 13

12. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

13. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit MISTAKES WERE MADE Thursday, May 16, 13

14. @isntitvacant NODEPDX 2013 wHY? 2-3 months ago js-git was funded

    wanted to “back with code” also to apply experience working on voxel.js Thursday, May 16, 13

15. @isntitvacant NODEPDX 2013 WHAT is git? Thursday, May 16, 13

16. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS Thursday, May 16,

    13

17. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TAGS Thursday, May

    16, 13

18. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS COMMITS TAGS Thursday,

    May 16, 13

19. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    Thursday, May 16, 13

20. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS Thursday, May 16, 13

21. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS object 171d82736413d1f781a16777bb67c270de9cc084 type commit tag v0.0.1 tagger Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ Sun Mar 24 00:10:34 2013 -0700 0.0.1 Thursday, May 16, 13

22. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS tree 27bae442f94a74dd7b070d8a37aa666665e6e123 parent dfa6cf3ebb37c53825035ca67482d3fc4c17e029 author Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ 1364109034 -0700 committer Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ 1364109034 -0700 0.0.1 Thursday, May 16, 13

23. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS 100644 blob f9fe4b03cd774c735d4eda711fbadc199c1f77af README.md 100644 blob 3ec1605cfba1dee1a31c76ccbefcd84b7a285d28 examples.js 100644 blob b5e785955ecd0329f6b771f32a7548f1a65e210a index.js 100644 blob 04e7de77ff4351c08fd37aa2079fd6f23a19e9c0 package.json Thursday, May 16, 13

24. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS { "name": "git-fs-repo", "version": "0.0.1", "description": "filesystem backed git repository", "main": "index.js", "scripts": { "test": "node test.js" }, ... Thursday, May 16, 13

25. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES Thursday, May

    16, 13

26. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES LOOSE Thursday,

    May 16, 13

27. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    Thursday, May 16, 13

28. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    .git/objects/{2 hex}/{38 hex} “commiT”, “BLOB”, “TAG”, “TREE” literal DECIMAL UNCOMPRESSED SIZE DEFLATED OBJECT DATA Thursday, May 16, 13

29. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    .git/objects/pack/{pack-hash}.idx .git/objects/pack/{pack-hash}.pack IDX =”index” random access for git objects in pack this is where all the deltas live more on packfiles later! Thursday, May 16, 13

30. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES var log

    = require('git-walk-refs') var repo = require('git-fs-repo') repo('path/to/.git', function(err, git) { var hashes = git.refs().map(function(ref) { return ref.hash }) log(git.find, hashes) .on('data', console.log) }) Thursday, May 16, 13

31. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT Thursday, May 16,

    13

32. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES Thursday, May

    16, 13

33. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 4 bytes:

    “PACK” 4 bytes: version 4 bytes: # of objs HEADER Thursday, May 16, 13

34. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 1 bit:

    continuation 3 bits: obj type 4 + N bits: obj size DEFLATED OBJ DATA PACKFILES Object Thursday, May 16, 13

35. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 20

    bytes: checksum checksum Thursday, May 16, 13

36. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 4

    valid types + 2 delta types “offset” and “reference” delta deltas are xdelta format reference has a hash, offset is an offset reference to a previous obj Thursday, May 16, 13

37. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... deltas

    may reference other deltas the packing heuristics are neat ... though totally optional Thursday, May 16, 13

38. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var fs

    = require('fs') var list = require('git-list-pack') var objectify = require('git-objectify-pack') var objects = {} var find = function(hash, ready) { return ready(null, objects[hash]) } fs.createReadStream('.git/objects/XXX.pack') .pipe(list()) .pipe(objectify(find)) .on('data', function(obj) { objects[obj.hash] = obj }) Thursday, May 16, 13

39. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn

    = net.connect({host: 'github.com', port: 9418}) var fetch = require('git-fetch-pack') var transport = require('git-transport-protocol') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/(master)/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(yourFavoriteKVStore()) Thursday, May 16, 13

40. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn

    = shoe.connect('/git') var fetch = require('git-fetch-pack') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/master/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(levelJS()) Thursday, May 16, 13

41. @isntitvacant NODEPDX 2013 Lessons. Thursday, May 16, 13

42. @isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

43. @isntitvacant NODEPDX 2013 write modules lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16,

    13

44. @isntitvacant NODEPDX 2013 write modules (LOTS OF small) lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO

    Thursday, May 16, 13

45. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

46. @isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

47. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

48. @isntitvacant NODEPDX 2013 lessons. easier to test Thursday, May 16,

    13

49. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    Thursday, May 16, 13

50. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    easier to document Thursday, May 16, 13

51. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    easier to document easier to change later Thursday, May 16, 13

52. @isntitvacant NODEPDX 2013 lessons. USE Thursday, May 16, 13

53. @isntitvacant NODEPDX 2013 lessons. USE github.com/substack/node-browserify Thursday, May 16, 13

54. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

55. @isntitvacant NODEPDX 2013 great build tool (with standalone) lessons. Thursday,

    May 16, 13

56. @isntitvacant NODEPDX 2013 great build tool (with standalone) great for

    shimming lessons. Thursday, May 16, 13

57. @isntitvacant NODEPDX 2013 great build tool (with standalone) great for

    shimming great tooling (source maps, beefy, testling ci) lessons. Thursday, May 16, 13

58. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

59. @isntitvacant NODEPDX 2013 lessons. put apis in terms of Thursday,

    May 16, 13

60. @isntitvacant NODEPDX 2013 lessons. put apis in terms of streams

    Thursday, May 16, 13

61. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

62. @isntitvacant NODEPDX 2013 lessons. pluggable api Thursday, May 16, 13

63. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test Thursday, May 16, 13

64. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process Thursday, May 16, 13

65. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process use through! Thursday, May 16, 13

66. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process use through! (and min-streams underneath) Thursday, May 16, 13

67. @isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

68. @isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

69. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson Thursday, May 16, 13

70. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant Thursday, May

    16, 13

71. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git

    / freenode Thursday, May 16, 13

72. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git

    / freenode and thanks to creationix, dominictarr, danlucraft, maksimlin, and others! Thursday, May 16, 13

73. @isntitvacant NODEPDX 2013 ISSUES. Thursday, May 16, 13

74. @isntitvacant NODEPDX 2013 lessons. ZLIB Thursday, May 16, 13

75. @isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

76. @isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s

    Thursday, May 16, 13

77. @isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s

    BUT you can’t get the unused bytes from inflate Thursday, May 16, 13

78. @isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

79. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate Thursday, May 16,

    13

80. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me

    sad Thursday, May 16, 13

81. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me

    sad but also happy! was fun. github.com/CHRISDICKINSON/inflate Thursday, May 16, 13

82. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

83. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

84. @isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere bu ff

    ers Thursday, May 16, 13

85. @isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere BUT typed

    arrays are better in the browser github.com/CHRISDICKINSON/bops bu ff ers Thursday, May 16, 13

86. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

87. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops bu ff ers Thursday, May

    16, 13

88. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    bu ff ers Thursday, May 16, 13

89. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    “bops” - buffer ops bu ff ers Thursday, May 16, 13

90. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    “bops” - buffer ops use typed arrays in browser, buffers elsewhere bu ff ers Thursday, May 16, 13

91. @isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

92. @isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

93. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult Thursday, May 16, 13

94. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it Thursday, May 16, 13

95. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it and bumping those deps Thursday, May 16, 13

96. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it and bumping those deps voxel.js ran into this with three.js Thursday, May 16, 13

97. @isntitvacant NODEPDX 2013 ISSUES. small modules Thursday, May 16, 13

98. @isntitvacant NODEPDX 2013 ISSUES. not insurmountable, though small modules Thursday,

    May 16, 13