JS-GIT

NodePDX 2013 JS-GIT talk

Transcript

1. @isntitvacant
   NODEPDX 2013
   Git
   JS
   Thursday, May 16, 13
   

   View Slide

2. @isntitvacant
   NODEPDX 2013
   Thursday, May 16, 13
   

   View Slide

3. @isntitvacant
   NODEPDX 2013
   HI.my name is chris.
   I work at .
   (it’s fun)
   (As a JS engineer) (We're hiring)
   Thursday, May 16, 13
   

   View Slide

4. @isntitvacant
   NODEPDX 2013
   wHY GIT?
   Thursday, May 16, 13
   

   View Slide

5. @isntitvacant
   NODEPDX 2013
   Made a GIT/HG site in
   2010 with a friend
   “repocracy”
   very nearly
   immediately trolled
   wHY?
   Thursday, May 16, 13
   

   View Slide

6. @isntitvacant
   NODEPDX 2013
   wHY?
   wHY?
   Thursday, May 16, 13
   

   View Slide

7. @isntitvacant
   NODEPDX 2013
   wHY?
   wHY?
   Thursday, May 16, 13
   

   View Slide

8. @isntitvacant
   NODEPDX 2013
   wHY?
   wHY?
   Thursday, May 16, 13
   

   View Slide

9. @isntitvacant
   NODEPDX 2013
   wHY?
   wanted to limit repo
   size before cloning
   hence, “tempisfugit”
   github.com/chrisdickinson/tempisfugit
   Thursday, May 16, 13
   

   View Slide

10. @isntitvacant
    NODEPDX 2013
    wHY?
    tempisfugit was awful,
    just awful
    no zlib in node yet
    heavily tied to fs api
    monolithic
    And, of course...
    github.com/chrisdickinson/tempisfugit
    Thursday, May 16, 13
    

    View Slide

11. @isntitvacant
    NODEPDX 2013
    fs.stat(baseFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(baseFol
    fs.stat(objectsFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(o
    fs.readFile(indexFile, function(err, indexDa
    err && exit(err, null) || (function() {
    fs.readFile(headFile, function(err, head
    err && exit(err, null) || (function()
    var odbObject = odb.ODB.init();
    exit(null, new Repository(odbObject,
    'repository':baseFolder,
    'index':indexFile,
    'head':headFile,
    'objects':objectsFolder,
    wHY?
    Thursday, May 16, 13
    

    View Slide

12. @isntitvacant
    NODEPDX 2013
    fs.stat(baseFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(baseFol
    fs.stat(objectsFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(o
    fs.readFile(indexFile, function(err, indexDa
    err && exit(err, null) || (function() {
    fs.readFile(headFile, function(err, head
    err && exit(err, null) || (function()
    var odbObject = odb.ODB.init();
    exit(null, new Repository(odbObject,
    'repository':baseFolder,
    'index':indexFile,
    'head':headFile,
    'objects':objectsFolder,
    wHY?
    github.com/chrisdickinson/tempisfugit
    Thursday, May 16, 13
    

    View Slide

13. @isntitvacant
    NODEPDX 2013
    fs.stat(baseFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(baseFol
    fs.stat(objectsFolder, function(err, stats) {
    err && exit(err, null) || (function() {
    !stats.isDirectory() && exit(errors.NOTAREPO(o
    fs.readFile(indexFile, function(err, indexDa
    err && exit(err, null) || (function() {
    fs.readFile(headFile, function(err, head
    err && exit(err, null) || (function()
    var odbObject = odb.ODB.init();
    exit(null, new Repository(odbObject,
    'repository':baseFolder,
    'index':indexFile,
    'head':headFile,
    'objects':objectsFolder,
    wHY?
    github.com/chrisdickinson/tempisfugit
    MISTAKES
    WERE MADE
    Thursday, May 16, 13
    

    View Slide

14. @isntitvacant
    NODEPDX 2013
    wHY?
    2-3 months ago
    js-git was funded
    wanted to “back
    with code”
    also to apply experience
    working on voxel.js
    Thursday, May 16, 13
    

    View Slide

15. @isntitvacant
    NODEPDX 2013
    WHAT is git?
    Thursday, May 16, 13
    

    View Slide

16. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    Thursday, May 16, 13
    

    View Slide

17. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TAGS
    Thursday, May 16, 13
    

    View Slide

18. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    COMMITS
    TAGS
    Thursday, May 16, 13
    

    View Slide

19. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    Thursday, May 16, 13
    

    View Slide

20. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    BLOBS
    Thursday, May 16, 13
    

    View Slide

21. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    BLOBS
    object 171d82736413d1f781a16777bb67c270de9cc084
    type commit
    tag v0.0.1
    tagger Chris Dickinson \
    \
    Sun Mar 24 00:10:34 2013 -0700
    0.0.1
    Thursday, May 16, 13
    

    View Slide

22. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    BLOBS
    tree 27bae442f94a74dd7b070d8a37aa666665e6e123
    parent dfa6cf3ebb37c53825035ca67482d3fc4c17e029
    author Chris Dickinson \
    \
    1364109034 -0700
    committer Chris Dickinson \
    \
    1364109034 -0700
    0.0.1
    Thursday, May 16, 13
    

    View Slide

23. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    BLOBS
    100644 blob f9fe4b03cd774c735d4eda711fbadc199c1f77af README.md
    100644 blob 3ec1605cfba1dee1a31c76ccbefcd84b7a285d28 examples.js
    100644 blob b5e785955ecd0329f6b771f32a7548f1a65e210a index.js
    100644 blob 04e7de77ff4351c08fd37aa2079fd6f23a19e9c0 package.json
    Thursday, May 16, 13
    

    View Slide

24. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    4OBJECTS
    TREES
    COMMITS
    TAGS
    BLOBS
    {
    "name": "git-fs-repo",
    "version": "0.0.1",
    "description": "filesystem backed git repository",
    "main": "index.js",
    "scripts": {
    "test": "node test.js"
    },
    ...
    Thursday, May 16, 13
    

    View Slide

25. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    Thursday, May 16, 13
    

    View Slide

26. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    LOOSE
    Thursday, May 16, 13
    

    View Slide

27. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    PACKFILES
    LOOSE
    Thursday, May 16, 13
    

    View Slide

28. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    PACKFILES
    LOOSE .git/objects/{2 hex}/{38 hex}
    “commiT”, “BLOB”, “TAG”, “TREE” literal
    DECIMAL UNCOMPRESSED SIZE
    DEFLATED OBJECT DATA
    Thursday, May 16, 13
    

    View Slide

29. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    PACKFILES
    LOOSE .git/objects/pack/{pack-hash}.idx
    .git/objects/pack/{pack-hash}.pack
    IDX =”index”
    random access for git objects in pack
    this is where all the deltas live
    more on packfiles later!
    Thursday, May 16, 13
    

    View Slide

30. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    2OBJECT STORES
    var log = require('git-walk-refs')
    var repo = require('git-fs-repo')
    repo('path/to/.git', function(err, git) {
    var hashes = git.refs().map(function(ref) {
    return ref.hash
    })
    log(git.find, hashes)
    .on('data', console.log)
    })
    Thursday, May 16, 13
    

    View Slide

31. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    Thursday, May 16, 13
    

    View Slide

32. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES
    Thursday, May 16, 13
    

    View Slide

33. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES 4 bytes: “PACK”
    4 bytes: version
    4 bytes: # of objs
    HEADER
    Thursday, May 16, 13
    

    View Slide

34. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES
    1 bit: continuation
    3 bits: obj type
    4 + N bits: obj size
    DEFLATED OBJ DATA
    PACKFILES
    Object
    Thursday, May 16, 13
    

    View Slide

35. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES ...
    20 bytes: checksum
    checksum
    Thursday, May 16, 13
    

    View Slide

36. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES
    ...
    4 valid types + 2 delta types
    “offset” and “reference” delta
    deltas are xdelta format
    reference has a hash, offset is
    an offset reference to a previous
    obj
    Thursday, May 16, 13
    

    View Slide

37. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    PACKFILES
    ...
    deltas may reference other
    deltas
    the packing heuristics are neat
    ... though totally optional
    Thursday, May 16, 13
    

    View Slide

38. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    ...
    var fs = require('fs')
    var list = require('git-list-pack')
    var objectify = require('git-objectify-pack')
    var objects = {}
    var find = function(hash, ready) {
    return ready(null, objects[hash])
    }
    fs.createReadStream('.git/objects/XXX.pack')
    .pipe(list())
    .pipe(objectify(find))
    .on('data', function(obj) { objects[obj.hash] = obj })
    Thursday, May 16, 13
    

    View Slide

39. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    ...
    var conn = net.connect({host: 'github.com', port: 9418})
    var fetch = require('git-fetch-pack')
    var transport = require('git-transport-protocol')
    var client
    client = fetch(
    'git://github.com/chrisdickinson/plate.git'
    , function want(ref, ready) {
    return ready(/(master)/.test(ref.name)) })
    )
    client
    .pipe(transport(conn))
    .pipe(client)
    client.pack
    .pipe(list())
    .pipe(objectify(find))
    .pipe(yourFavoriteKVStore())
    Thursday, May 16, 13
    

    View Slide

40. @isntitvacant
    NODEPDX 2013
    WHAT?
    GIT IS
    1TRANSPORT
    ...
    var conn = shoe.connect('/git')
    var fetch = require('git-fetch-pack')
    var client
    client = fetch(
    'git://github.com/chrisdickinson/plate.git'
    , function want(ref, ready) {
    return ready(/master/.test(ref.name)) })
    )
    client
    .pipe(transport(conn))
    .pipe(client)
    client.pack
    .pipe(list())
    .pipe(objectify(find))
    .pipe(levelJS())
    Thursday, May 16, 13
    

    View Slide

41. @isntitvacant
    NODEPDX 2013
    Lessons.
    Thursday, May 16, 13
    

    View Slide

42. @isntitvacant
    NODEPDX 2013
    lessons.
    github.com/CHRISDICKINSON/JS-GIT-DEMO
    Thursday, May 16, 13
    

    View Slide

43. @isntitvacant
    NODEPDX 2013
    write
    modules
    lessons.
    github.com/CHRISDICKINSON/JS-GIT-DEMO
    Thursday, May 16, 13
    

    View Slide

44. @isntitvacant
    NODEPDX 2013
    write
    modules
    (LOTS OF small)
    lessons.
    github.com/CHRISDICKINSON/JS-GIT-DEMO
    Thursday, May 16, 13
    

    View Slide

45. @isntitvacant
    NODEPDX 2013
    lessons.
    Thursday, May 16, 13
    

    View Slide

46. @isntitvacant
    NODEPDX 2013
    lessons.
    github.com/CHRISDICKINSON/JS-GIT-DEMO
    Thursday, May 16, 13
    

    View Slide

47. @isntitvacant
    NODEPDX 2013
    lessons.
    Thursday, May 16, 13
    

    View Slide

48. @isntitvacant
    NODEPDX 2013
    lessons.
    easier to test
    Thursday, May 16, 13
    

    View Slide

49. @isntitvacant
    NODEPDX 2013
    lessons.
    easier to test
    easier to write
    Thursday, May 16, 13
    

    View Slide

50. @isntitvacant
    NODEPDX 2013
    lessons.
    easier to test
    easier to write
    easier to document
    Thursday, May 16, 13
    

    View Slide

51. @isntitvacant
    NODEPDX 2013
    lessons.
    easier to test
    easier to write
    easier to document
    easier to change later
    Thursday, May 16, 13
    

    View Slide

52. @isntitvacant
    NODEPDX 2013
    lessons.
    USE
    Thursday, May 16, 13
    

    View Slide

53. @isntitvacant
    NODEPDX 2013
    lessons.
    USE
    github.com/substack/node-browserify
    Thursday, May 16, 13
    

    View Slide

54. @isntitvacant
    NODEPDX 2013
    lessons.
    Thursday, May 16, 13
    

    View Slide

55. @isntitvacant
    NODEPDX 2013
    great build tool (with
    standalone)
    lessons.
    Thursday, May 16, 13
    

    View Slide

56. @isntitvacant
    NODEPDX 2013
    great build tool (with
    standalone)
    great for shimming
    lessons.
    Thursday, May 16, 13
    

    View Slide

57. @isntitvacant
    NODEPDX 2013
    great build tool (with
    standalone)
    great for shimming
    great tooling (source
    maps, beefy, testling ci)
    lessons.
    Thursday, May 16, 13
    

    View Slide

58. @isntitvacant
    NODEPDX 2013
    lessons.
    Thursday, May 16, 13
    

    View Slide

59. @isntitvacant
    NODEPDX 2013
    lessons.
    put apis in terms of
    Thursday, May 16, 13
    

    View Slide

60. @isntitvacant
    NODEPDX 2013
    lessons.
    put apis in terms of
    streams
    Thursday, May 16, 13
    

    View Slide

61. @isntitvacant
    NODEPDX 2013
    lessons.
    Thursday, May 16, 13
    

    View Slide

62. @isntitvacant
    NODEPDX 2013
    lessons.
    pluggable api
    Thursday, May 16, 13
    

    View Slide

63. @isntitvacant
    NODEPDX 2013
    lessons.
    pluggable api
    don’t need the entire
    thing working to test
    Thursday, May 16, 13
    

    View Slide

64. @isntitvacant
    NODEPDX 2013
    lessons.
    pluggable api
    don’t need the entire
    thing working to test
    don’t need all of the
    data at once to process
    Thursday, May 16, 13
    

    View Slide

65. @isntitvacant
    NODEPDX 2013
    lessons.
    pluggable api
    don’t need the entire
    thing working to test
    don’t need all of the
    data at once to process
    use through!
    Thursday, May 16, 13
    

    View Slide

66. @isntitvacant
    NODEPDX 2013
    lessons.
    pluggable api
    don’t need the entire
    thing working to test
    don’t need all of the
    data at once to process
    use through! (and min-streams underneath)
    Thursday, May 16, 13
    

    View Slide

67. @isntitvacant
    NODEPDX 2013
    thanks.
    Thursday, May 16, 13
    

    View Slide

68. @isntitvacant
    NODEPDX 2013
    thanks.
    Thursday, May 16, 13
    

    View Slide

69. @isntitvacant
    NODEPDX 2013
    thanks.
    github: chrisdickinson
    Thursday, May 16, 13
    

    View Slide

70. @isntitvacant
    NODEPDX 2013
    thanks.
    github: chrisdickinson
    twitter: @isntitvacant
    Thursday, May 16, 13
    

    View Slide

71. @isntitvacant
    NODEPDX 2013
    thanks.
    github: chrisdickinson
    twitter: @isntitvacant
    irc: #js-git / freenode
    Thursday, May 16, 13
    

    View Slide

72. @isntitvacant
    NODEPDX 2013
    thanks.
    github: chrisdickinson
    twitter: @isntitvacant
    irc: #js-git / freenode
    and thanks to creationix, dominictarr,
    danlucraft, maksimlin, and others!
    Thursday, May 16, 13
    

    View Slide

73. @isntitvacant
    NODEPDX 2013
    ISSUES.
    Thursday, May 16, 13
    

    View Slide

74. @isntitvacant
    NODEPDX 2013
    lessons.
    ZLIB
    Thursday, May 16, 13
    

    View Slide

75. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    Thursday, May 16, 13
    

    View Slide

76. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    there’s a shim + node’s
    Thursday, May 16, 13
    

    View Slide

77. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    there’s a shim + node’s
    BUT you can’t get the
    unused bytes from
    inflate
    Thursday, May 16, 13
    

    View Slide

78. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    Thursday, May 16, 13
    

    View Slide

79. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    rewrote inflate
    Thursday, May 16, 13
    

    View Slide

80. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    rewrote inflate
    this makes me sad
    Thursday, May 16, 13
    

    View Slide

81. @isntitvacant
    NODEPDX 2013
    ISSUES.
    ZLIB
    rewrote inflate
    this makes me sad
    but also happy! was fun.
    github.com/CHRISDICKINSON/inflate
    Thursday, May 16, 13
    

    View Slide

82. @isntitvacant
    NODEPDX 2013
    ISSUES.
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

83. @isntitvacant
    NODEPDX 2013
    ISSUES.
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

84. @isntitvacant
    NODEPDX 2013
    ISSUES.
    browserify’s shim works
    everywhere
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

85. @isntitvacant
    NODEPDX 2013
    ISSUES.
    browserify’s shim works
    everywhere
    BUT typed arrays are
    better in the browser
    github.com/CHRISDICKINSON/bops
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

86. @isntitvacant
    NODEPDX 2013
    ISSUES.
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

87. @isntitvacant
    NODEPDX 2013
    ISSUES.
    github.com/CHRISDICKINSON/bops
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

88. @isntitvacant
    NODEPDX 2013
    ISSUES.
    github.com/CHRISDICKINSON/bops
    use a different primitive
    module
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

89. @isntitvacant
    NODEPDX 2013
    ISSUES.
    github.com/CHRISDICKINSON/bops
    use a different primitive
    module
    “bops” - buffer ops
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

90. @isntitvacant
    NODEPDX 2013
    ISSUES.
    github.com/CHRISDICKINSON/bops
    use a different primitive
    module
    “bops” - buffer ops
    use typed arrays in
    browser, buffers
    elsewhere
    bu
    ff
    ers
    Thursday, May 16, 13
    

    View Slide

91. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    Thursday, May 16, 13
    

    View Slide

92. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    Thursday, May 16, 13
    

    View Slide

93. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    using a different primitive is
    difficult
    Thursday, May 16, 13
    

    View Slide

94. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    using a different primitive is
    difficult
    requires making all modules dep
    on it
    Thursday, May 16, 13
    

    View Slide

95. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    using a different primitive is
    difficult
    requires making all modules dep
    on it
    and bumping those deps
    Thursday, May 16, 13
    

    View Slide

96. @isntitvacant
    NODEPDX 2013
    small
    modules
    ISSUES.
    using a different primitive is
    difficult
    requires making all modules dep
    on it
    and bumping those deps
    voxel.js ran into this with
    three.js
    Thursday, May 16, 13
    

    View Slide

97. @isntitvacant
    NODEPDX 2013
    ISSUES.
    small
    modules
    Thursday, May 16, 13
    

    View Slide

98. @isntitvacant
    NODEPDX 2013
    ISSUES.
    not insurmountable, though
    small
    modules
    Thursday, May 16, 13
    

    View Slide