$30 off During Our Annual Pro Sale. View Details »

JS-GIT

Chris Dickinson
May 16, 2013
Programming
7
390

 JS-GIT

NodePDX 2013 JS-GIT talk

Chris Dickinson

May 16, 2013
Tweet

More Decks by Chris Dickinson

See All by Chris Dickinson
Better Living Through Control Flow Graph Generation
chrisdickinson
2
130
My Little Parser -- Tooling is Magic
chrisdickinson
0
510
JS at Urban Airship
chrisdickinson
4
430

Other Decks in Programming

See All in Programming
Kubernetes Admission Webhook Deep Dive
zoetrope
7
640
Micro Frontends with Module Federation: Beyond the Basics
manfredsteyer
PRO
0
310
Blue / Green デプロイと安全性と複雑性と #AWSDevDay
track3jyo
PRO
11
3.9k
Yet Another KMP*
ilkeraslan
0
110
よくあるケースから学ぶ！AWS で想定外のコスト増につながるような実装や設計
track3jyo
PRO
9
640
脱Firebase. 我々はどう生きるか/Migrate from Firebase
keisuke69
8
4.2k
GDG DevFest2022 Songdo - KMM(Kotlin Multiplatform Mobile)
taehwandev
0
240
Behat en 2022
pocky
1
150
Elm_LandでElm入門
negiboudu
0
150
なんでもPull Requestにする / Kichijoji.pm 31
utgwkk
1
1.4k
Django ORM パフォーマンスTips
takeaship
0
230
Amazon S3 Selectで実現するサーバーレス高負荷対応サイト
seike460
PRO
2
570

Featured

See All Featured
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.7k
Rebuilding a faster, lazier Slack
samanthasiow
65
7.5k
Building a Modern Day 
E-commerce SEO Strategy
aleyda
4
3.8k
Scaling GitHub
holman
453
140k
Visualization
eitanlees
126
12k
No one is an island. Learnings from fostering a developers community.
thoeni
11
1.5k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
101
6.1k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
20
6.6k
Design by the Numbers
sachag
271
17k
What’s in a name? Adding method to the madness
productmarketing
11
1.8k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
12
1.1k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
657
120k

Transcript

  1. @isntitvacant NODEPDX 2013 Git JS Thursday, May 16, 13

  2. @isntitvacant NODEPDX 2013 Thursday, May 16, 13

  3. @isntitvacant NODEPDX 2013 HI.my name is chris. I work at

    . (it’s fun) (As a JS engineer) (We're hiring) Thursday, May 16, 13

  4. @isntitvacant NODEPDX 2013 wHY GIT? Thursday, May 16, 13

  5. @isntitvacant NODEPDX 2013 Made a GIT/HG site in 2010 with

    a friend “repocracy” very nearly immediately trolled wHY? Thursday, May 16, 13

  6. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

  7. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

  8. @isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

  9. @isntitvacant NODEPDX 2013 wHY? wanted to limit repo size before

    cloning hence, “tempisfugit” github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

  10. @isntitvacant NODEPDX 2013 wHY? tempisfugit was awful, just awful no

    zlib in node yet heavily tied to fs api monolithic And, of course... github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

  11. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? Thursday, May 16, 13

  12. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

  13. @isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,

    null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit MISTAKES WERE MADE Thursday, May 16, 13

  14. @isntitvacant NODEPDX 2013 wHY? 2-3 months ago js-git was funded

    wanted to “back with code” also to apply experience working on voxel.js Thursday, May 16, 13

  15. @isntitvacant NODEPDX 2013 WHAT is git? Thursday, May 16, 13

  16. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS Thursday, May 16,

    13

  17. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TAGS Thursday, May

    16, 13

  18. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS COMMITS TAGS Thursday,

    May 16, 13

  19. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    Thursday, May 16, 13

  20. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS Thursday, May 16, 13

  21. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS object 171d82736413d1f781a16777bb67c270de9cc084 type commit tag v0.0.1 tagger Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ Sun Mar 24 00:10:34 2013 -0700 0.0.1 Thursday, May 16, 13

  22. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS tree 27bae442f94a74dd7b070d8a37aa666665e6e123 parent dfa6cf3ebb37c53825035ca67482d3fc4c17e029 author Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ 1364109034 -0700 committer Chris Dickinson \ <christopher.s.dickinson@gmail.com> \ 1364109034 -0700 0.0.1 Thursday, May 16, 13

  23. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS 100644 blob f9fe4b03cd774c735d4eda711fbadc199c1f77af README.md 100644 blob 3ec1605cfba1dee1a31c76ccbefcd84b7a285d28 examples.js 100644 blob b5e785955ecd0329f6b771f32a7548f1a65e210a index.js 100644 blob 04e7de77ff4351c08fd37aa2079fd6f23a19e9c0 package.json Thursday, May 16, 13

  24. @isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS

    BLOBS { "name": "git-fs-repo", "version": "0.0.1", "description": "filesystem backed git repository", "main": "index.js", "scripts": { "test": "node test.js" }, ... Thursday, May 16, 13

  25. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES Thursday, May

    16, 13

  26. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES LOOSE Thursday,

    May 16, 13

  27. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    Thursday, May 16, 13

  28. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    .git/objects/{2 hex}/{38 hex} “commiT”, “BLOB”, “TAG”, “TREE” literal DECIMAL UNCOMPRESSED SIZE DEFLATED OBJECT DATA Thursday, May 16, 13

  29. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

    .git/objects/pack/{pack-hash}.idx .git/objects/pack/{pack-hash}.pack IDX =”index” random access for git objects in pack this is where all the deltas live more on packfiles later! Thursday, May 16, 13

  30. @isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES var log

    = require('git-walk-refs') var repo = require('git-fs-repo') repo('path/to/.git', function(err, git) { var hashes = git.refs().map(function(ref) { return ref.hash }) log(git.find, hashes) .on('data', console.log) }) Thursday, May 16, 13

  31. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT Thursday, May 16,

    13

  32. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES Thursday, May

    16, 13

  33. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 4 bytes:

    “PACK” 4 bytes: version 4 bytes: # of objs HEADER Thursday, May 16, 13

  34. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 1 bit:

    continuation 3 bits: obj type 4 + N bits: obj size DEFLATED OBJ DATA PACKFILES Object Thursday, May 16, 13

  35. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 20

    bytes: checksum checksum Thursday, May 16, 13

  36. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 4

    valid types + 2 delta types “offset” and “reference” delta deltas are xdelta format reference has a hash, offset is an offset reference to a previous obj Thursday, May 16, 13

  37. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... deltas

    may reference other deltas the packing heuristics are neat ... though totally optional Thursday, May 16, 13

  38. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var fs

    = require('fs') var list = require('git-list-pack') var objectify = require('git-objectify-pack') var objects = {} var find = function(hash, ready) { return ready(null, objects[hash]) } fs.createReadStream('.git/objects/XXX.pack') .pipe(list()) .pipe(objectify(find)) .on('data', function(obj) { objects[obj.hash] = obj }) Thursday, May 16, 13

  39. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn

    = net.connect({host: 'github.com', port: 9418}) var fetch = require('git-fetch-pack') var transport = require('git-transport-protocol') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/(master)/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(yourFavoriteKVStore()) Thursday, May 16, 13

  40. @isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn

    = shoe.connect('/git') var fetch = require('git-fetch-pack') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/master/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(levelJS()) Thursday, May 16, 13

  41. @isntitvacant NODEPDX 2013 Lessons. Thursday, May 16, 13

  42. @isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

  43. @isntitvacant NODEPDX 2013 write modules lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16,

    13

  44. @isntitvacant NODEPDX 2013 write modules (LOTS OF small) lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO

    Thursday, May 16, 13

  45. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

  46. @isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

  47. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

  48. @isntitvacant NODEPDX 2013 lessons. easier to test Thursday, May 16,

    13

  49. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    Thursday, May 16, 13

  50. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    easier to document Thursday, May 16, 13

  51. @isntitvacant NODEPDX 2013 lessons. easier to test easier to write

    easier to document easier to change later Thursday, May 16, 13

  52. @isntitvacant NODEPDX 2013 lessons. USE Thursday, May 16, 13

  53. @isntitvacant NODEPDX 2013 lessons. USE github.com/substack/node-browserify Thursday, May 16, 13

  54. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

  55. @isntitvacant NODEPDX 2013 great build tool (with standalone) lessons. Thursday,

    May 16, 13

  56. @isntitvacant NODEPDX 2013 great build tool (with standalone) great for

    shimming lessons. Thursday, May 16, 13

  57. @isntitvacant NODEPDX 2013 great build tool (with standalone) great for

    shimming great tooling (source maps, beefy, testling ci) lessons. Thursday, May 16, 13

  58. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

  59. @isntitvacant NODEPDX 2013 lessons. put apis in terms of Thursday,

    May 16, 13

  60. @isntitvacant NODEPDX 2013 lessons. put apis in terms of streams

    Thursday, May 16, 13

  61. @isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

  62. @isntitvacant NODEPDX 2013 lessons. pluggable api Thursday, May 16, 13

  63. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test Thursday, May 16, 13

  64. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process Thursday, May 16, 13

  65. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process use through! Thursday, May 16, 13

  66. @isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire

    thing working to test don’t need all of the data at once to process use through! (and min-streams underneath) Thursday, May 16, 13

  67. @isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

  68. @isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

  69. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson Thursday, May 16, 13

  70. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant Thursday, May

    16, 13

  71. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git

    / freenode Thursday, May 16, 13

  72. @isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git

    / freenode and thanks to creationix, dominictarr, danlucraft, maksimlin, and others! Thursday, May 16, 13

  73. @isntitvacant NODEPDX 2013 ISSUES. Thursday, May 16, 13

  74. @isntitvacant NODEPDX 2013 lessons. ZLIB Thursday, May 16, 13

  75. @isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

  76. @isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s

    Thursday, May 16, 13

  77. @isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s

    BUT you can’t get the unused bytes from inflate Thursday, May 16, 13

  78. @isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

  79. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate Thursday, May 16,

    13

  80. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me

    sad Thursday, May 16, 13

  81. @isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me

    sad but also happy! was fun. github.com/CHRISDICKINSON/inflate Thursday, May 16, 13

  82. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

  83. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

  84. @isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere bu ff

    ers Thursday, May 16, 13

  85. @isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere BUT typed

    arrays are better in the browser github.com/CHRISDICKINSON/bops bu ff ers Thursday, May 16, 13

  86. @isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,

    13

  87. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops bu ff ers Thursday, May

    16, 13

  88. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    bu ff ers Thursday, May 16, 13

  89. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    “bops” - buffer ops bu ff ers Thursday, May 16, 13

  90. @isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module

    “bops” - buffer ops use typed arrays in browser, buffers elsewhere bu ff ers Thursday, May 16, 13

  91. @isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

  92. @isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

  93. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult Thursday, May 16, 13

  94. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it Thursday, May 16, 13

  95. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it and bumping those deps Thursday, May 16, 13

  96. @isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive

    is difficult requires making all modules dep on it and bumping those deps voxel.js ran into this with three.js Thursday, May 16, 13

  97. @isntitvacant NODEPDX 2013 ISSUES. small modules Thursday, May 16, 13

  98. @isntitvacant NODEPDX 2013 ISSUES. not insurmountable, though small modules Thursday,

    May 16, 13