Internals of Git Internals

Transcript

1. Internals of Git Internals Minqi Pan

2. I’m Minqi Pan github.com/pmq20 twitter @psvr

3. Synopsis • Internals of downloading code • Internals of uploading

   code

4. Question:
 Which commands to use?

5. 140 commands add, am, annotate, apply, archimport, archive, bisect, blame,

   branch, bundle, cat-file, check-attr, check-ignore, check-mailmap, checkout, checkout-index, check-ref-format, cherry, cherry-pick, citool, clean, clone, column, commit, commit-tree, config, count- objects, credential, credential-cache, credential-store, cvsexportcommit, cvsimport, cvsserver, daemon, describe, diff, diff-files, diff-index, diff-tree, difftool, fast-export, fast-import, fetch, fetch-pack, filter-branch, fmt-merge-msg, for-each-ref, format-patch, fsck, gc, get-tar-commit-id, grep, gui, hash-object, help, http-backend, http-fetch, http- push, imap-send, index-pack, init, instaweb, interpret-trailers, log, ls-files, ls-remote, ls- tree, mailinfo, mailsplit, merge, merge-base, merge-file, merge-index, merge-one-file, mergetool, merge-tree, mktag, mktree, mv, name-rev, notes, p4, pack-objects, pack- redundant, pack-refs, parse-remote, patch-id, prune, prune-packed, pull, push, quiltimport, read-tree, rebase, receive-pack, reflog, relink, remote, repack, replace, request-pull, rerere, reset, revert, rev-list, rev-parse, rm, send-email, send-pack, shell, shortlog, show, show-branch, show-index, show-ref, sh-i18n, sh-setup, stash, status, stripspace, submodule, svn, symbolic-ref, tag, unpack-file, unpack-objects, update- index, update-ref, update-server-info, upload-archive, upload-pack, var, verify-commit, verify-pack, verify-tag, whatchanged, worktree, write-tree

6. – Chapter 10. Git Internals “Git is fundamentally a content-addressable

   filesystem with a VCS user interface written on top of it.”

7. a content-addressable filesystem

8. simple
 straightforward
 a whole-object store

9. “Let there be packs!”

10. Pack-files • USAGE I: streaming • USAGE II: on-disk storage

    • w/ nice balance of density vs ease-of use • “Git really doesn't follow files”

11. Git Objects Types

12. Random Access • compressed just one object at a time

    • pessimizations due to double usage: low compression factor as a whole • able to translate object name to location in pack

13. Spotlights of today add, am, annotate, apply, archimport, archive, bisect,

    blame, branch, bundle, cat-file, check-attr, check-ignore, check-mailmap, checkout, checkout-index, check-ref-format, cherry, cherry-pick, citool, clean, clone, column, commit, commit-tree, config, count-objects, credential, credential- cache, credential-store, cvsexportcommit, cvsimport, cvsserver, daemon, describe, diff, diff-files, diff-index, diff-tree, difftool, fast-export, fast-import, fetch, fetch-pack, filter-branch, fmt- merge-msg, for-each-ref, format-patch, fsck, gc, get-tar-commit-id, grep, gui, hash-object, help, http-backend, http-fetch, http-push, imap-send, index-pack, init, instaweb, interpret-trailers, log, ls-files, ls-remote, ls-tree, mailinfo, mailsplit, merge, merge-base, merge-file, merge-index, merge-one-file, mergetool, merge-tree, mktag, mktree, mv, name-rev, notes, p4, pack- objects, pack-redundant, pack-refs, parse-remote, patch-id, prune, prune-packed, pull, push, quiltimport, read-tree, rebase, receive-pack, reflog, relink, remote, repack, replace, request-pull, rerere, reset, revert, rev-list, rev-parse, rm, send-email, send-pack, shell, shortlog, show, show-branch, show-index, show-ref, sh-i18n, sh-setup, stash, status, stripspace, submodule, svn, symbolic-ref, tag, unpack-file, unpack-objects, update-index, update-ref, update-server-info, upload-archive, upload-pack, var, verify-commit, verify-pack, verify-tag, whatchanged, worktree, write-tree

14. Remote Procedure Calls • git-fetch-pack —ssh,http—> git-upload-pack • git-send-pack —ssh,http—>

    git-receive-pack

15. Local UNIX-y Calls • git-upload-pack calls git-pack-objects • git-receive-pack calls

    git-unpack-objects • git-receive-pack also calls git-index-pack

16. Advantage of being UNIX-y Use of global variables anywhere Needless

    to pass around

17. git-upload-pack • first called with --stateless-rpc --advertise-refs • then called

    with --stateless-rpc • internally calls git-pack-objects

18. git-upload-pack --advertise-refs 001e# service=git-upload-pack 000000d10ffaa5a6fdf84714deddaecc62edd5540e0f5877 HEAD multi_ack thin- pack side-band

    side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/master agent=git/2.2.2 0057a576d8e0d128b503bb3d9980387fea0976ab6921 refs/heads/bug_fix/a 0053affee7f8adf11c7d895189ec5f0c3aeb95dba01a refs/heads/bug_fix/b 0053cbe2e212b415f587f086af6a74670919ccb7b089 refs/heads/bug_fix/c 004568df60003126a1f6478eead611af1c9000dfe44e refs/heads/feature/ci 0052c063094d4f94e182a23aa9a479bf67bf6e2de6b4 refs/heads/feature/d 004cfd02d43bc2ca02d381275453383e2c2629e33da5 refs/heads/feature/e 0049fdb1e470fd4d9be344dc77645bf777eec07d7940 refs/heads/feature/f … 0000

19. git-pack-objects • /usr/bin/git • pack-objects • --revs • --stdout •

    --progress • --delta-base-offset

20. pack-*.idx a simple index to do "object name -> location

    in pack-files” translations

21. pack-*.idx Data Structure

22. pack-*.idx Data Structure

23. pack-*.idx Data Structure A 256-entry fan-out table metaphorically 256 *

    4-byte

24. pack-*.idx Data Structure 0x0309 == number of objects whose first

    byte of object name is less than or equal to 1 0x060D == number of objects whose first byte of object name is less than or equal to 2 …

25. pack-*.idx Data Structure Question: Why 256 entries in total?

26. pack-*.idx Data Structure Answer: size of a byte of the

    first 20 bytes of SHA-1

27. pack-*.idx Data Structure Question: Which byte has the number of

    total objects?

28. pack-*.idx Data Structure Answer: 8 + 255 * 4 ~

    8 + 256 * 4 1028 ~ 1032

29. pack-*.idx Data Structure Read it: 0x02f1f2 == 193010

30. pack-*.idx Data Structure sorted 20-byte SHA-1 object names

31. pack-*.idx Data Structure Question: Why Sorted?

32. pack-*.idx Data Structure Answer: Binary search after fan-out

33. pack-*.idx Data Structure Question: Ends where?

34. pack-*.idx Data Structure Answer: 1032 + 20 * 0x02f1f2
 ==


    3861232

35. pack-*.idx Data Structure A table of 4-byte CRC32 values of

    the packed object data.

36. pack-*.idx Data Structure Question: Ends where?

37. pack-*.idx Data Structure Answer: 3861232 + 4 * 0x02f1f2 ==

    4633272

38. pack-*.idx Data Structure 4-byte offset values MSB + 31-bit Most

    Significant Bit marks the usage of 8-byte table

39. pack-*.idx Data Structure Question: Any limitations?

40. pack-*.idx Data Structure Answer: Pack files can only be 2147483648

    in size at best

41. pack-*.idx Data Structure Question: Ends where?

42. pack-*.idx Data Structure Answer: 4633272 + 4 * 0x02f1f2 ==

    5405312

43. pack-*.idx Data Structure A table of 8-byte offset entries. (empty

    for pack files less than 2 GiB) Question: Any limitations?

44. pack-*.idx Data Structure A table of 8-byte offset entries. (empty

    for pack files less than 2 GiB) Answer: No. What file on earth weights 16800000 TB? Well supports packs larger than 2 GiB

45. pack-*.idx Data Structure (trailer) A copy of the 20-byte SHA-1

    checksum at the end of corresponding packfile. 20-byte SHA-1-checksum of all of the above.

46. pack-*.pack Data Structure header

47. pack-*.pack Data Structure

48. pack-*.pack Data Structure

49. pack-*.pack Data Structure

50. pack-*.pack Data Structure n * object entries

51. Example • First byte of the name is 0x9f •

    IDX[8 + (0x9f - 1) * 4] == 0x0403 == 1027 • IDX[8 + 0x9f * 4] == 0x0403 == 1029 • Object No. 1027 ~ 1029 Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9

52. Example • Binary search 1027 ~ 1029 • Found at

    8 + 4 * 256 + 1027 * 20 == 21572 • Skip the rest total_num*(20+4) == 1628*24 Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9

53. Example • IDX[8 + 4 * 256 + 1628*24 +

    4 * 1027] Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 • PACK[0x0004482D] == PACK[280621]

54. Example Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 E3 11100011 1_______ => MSB 1 continue

    _110____ => type == 6 == OFS_DELTA ____0011 => length == 3 3-bit type, (n-1)*7+4-bit length

55. Example Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 • PACK[0x0004482D] 01 00000001 0_______ => MSB

    0 break _0000001 => length += (1 << 4) final length == 19

56. Example Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 • PACK[0x0004482D] AA 10101010 1_______ MSB 1

    continue _0101010 base offset == 42

57. Example Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 • PACK[0x0004482D] 44 01000100 0_______ MSB 0

    break _1000100 offset == ((42+1)<<7)+68 == 5572

58. Example Read 9fcf811e00fa469688943a9152c16d4ee90fb9a9 offset == 5572 push 0x0004482D into stack

    deal with (0x0004482D - 5572) push (0x0004482D - 5572) into stack … root base

59. Example SHA1 type size size-pack offset- pack depth base 9fcf811e00fa469

    688943a9152c16d 4ee90fb9a9 blob 19 32 280621 4 6110c89446f2281 e5db9b798a0fa02 0fad6e63e1 6110c89446f2281 e5db9b798a0fa02 0fad6e63e1 blob 52 45 275049 3 3bbeff3fc22b75c 1a26f4ab9b64449 b33002aea5 3bbeff3fc22b75c 1a26f4ab9b64449 b33002aea5 blob 2935 1263 273786 2 a39920830904665 6ecc01f7653c5d5 b8905fc16e a39920830904665 6ecc01f7653c5d5 b8905fc16e blob 4686 1540 272246 1 e4e56117de8b3bd 0bd899701da4712 caee27c7d6 e4e56117de8b3bd 0bd899701da4712 caee27c7d6 blob 12635 3279 115703 0 -

60. – Linus Torvalds “I played around with different delta algorithms,

    and with making the delta window bigger, but having too big of a sliding window makes it very expensive to generate the pack: you need to compare every object with a _ton_ of other objects.”

61. – Linus Torvalds “ANY order will give you a working

    pack, ... [but it is] the thing that gives packs good locality. It keeps the objects close to the head (whether they are old or new, but they are _reachable_ from the head) at the head of the pack. So packs actually have absolutely _wonderful_ IO patterns.”

62. Packing Heuristics • First sort by delta order • Then

    sort by recency order

63. Delta Order Heuristics • first sort by type. Different objects

    never delta with each other. • we do not delta different object types.

64. Delta Order Heuristics • then sort by filename/dirname. • we

    prefer to delta the objects with the same full path, but allow files with the same name from different directories.

65. Delta Order Heuristics • then if we are doing "thin"

    pack, the objects we are _not_ going to pack but we know about are sorted earlier than other objects. • we always prefer to delta against objects we are not going to send, if there are some. • for "thin" packs only. used when the other side is known to have such objects.

66. Delta Order Heuristics • and finally sort by size, larger

    to smaller. • we prefer to delta against larger objects, so that we have lots of removals. • large->small matters because of compression behaviour.

67. sort2: recency sort 1: (type, basename, size)

68. Reconciling Two Sorts • Linus' law: files grow, larger objects

    tend to be "more recent" • we only write out the base object first if the delta against it was more recent • Thus the front of the pack always contains data that is relevant to a “recent" object

69. As a Result • delta order and recency order match

    each other quite well • xdelta, removing data is cheaper (in size) than adding data • xdelta, larger->small is actually a big space saver too

70. git-pack-objects w/ threads --threads=<n> Specifies the number of threads to

    spawn when searching for best delta matches. This requires that pack-objects be compiled with pthreads otherwise this option is ignored with a warning. This is meant to reduce packing time on multiprocessor machines. The required amount of memory for the delta search window is however multiplied by the number of threads. Specifying 0 will cause Git to auto-detect the number of CPU's and set the number of threads accordingly.

71. git-receive-pack • First called with --advertise-refs • (ntohl(hdr.hdr_entries) < unpack_limit)

    =>
 git-unpack-objects • (ntohl(hdr.hdr_entries) >= unpack_limit) =>
 git-index-pack • Finally update refs with locks

72. Be safe

73. Be safe

74. Update refs with lock

75. Thank you https://github.com/pmq20/