Git LFS at Light Speed

Git and its extensions are becoming more popular than ever. However, certain use cases may still be suboptimal. We identified a way to dramatically improve performance in a popular Git extension - LFS (Large File Storage) - that required changes in both Git Core, and the extension itself. Based on our combined experiences we’ll walk you through the process of a successful contribution to each project with the help of mailing lists and Pull Requests. If you have already a bit of Git command line knowledge then this talk will prepare you for your first contribution to Git, an extension, or both.

Presented at GitMerge 2017 by Taylor Blau (GitHub) and Lars Schneider (Autodesk).

Transcript

1. GIT LFS @ LIGHT SPEED - The story of a Git contribution - Lars Schneider Autodesk Taylor Blau GitHub
2. @kit3bus Previous version Current version GIT LFS @ LIGHT SPEED
3. @kit3bus Recap: Why Git LFS*? * Git Large File Storage = 100 MB video file
4. @kit3bus Clone: Git without LFS 0 MB
5. @kit3bus Clone: Git without LFS 101 MB master 1 MB 100 MB
6. @kit3bus Clone: Git without LFS 201 MB master + 100 MB 1 MB 100 MB 1 MB 100 MB
7. @kit3bus Clone: Git without LFS master 301MB + 100 MB + 100 MB 1 MB 100 MB 1 MB 100 MB 1 MB 100 MB
8. @kit3bus Clone: Git with LFS 0 MB
9. @kit3bus LFS Server Clone: Git with LFS 1MB master 1 MB 100 MB 0 MB
10. @kit3bus LFS Server Clone: Git with LFS 1MB master 1 MB 100 MB 1 MB 100 MB 0 MB 0 MB
11. @kit3bus LFS Server Clone: Git with LFS 1MB master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB
12. @kit3bus LFS Server Git with LFS master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB Clone: 1 MB
13. @kit3bus LFS Server Git with LFS master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB Clone: 1 MB + Checkout: 100 MB
14. @kit3bus LFS Server Git with LFS master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB Clone: 1 MB + Checkout: 100 MB 60% bandwidth saving!
15. @kit3bus Git Checkout Git
16. @kit3bus Git Checkout Git Git LFS Git clean / smudge filter
17. @kit3bus Git Git Checkout Git LFS Git LFS
18. @kit3bus Git LFS Git LFS Git LFS One-Shot Filter: Many Git LFS invocations Git Checkout Git
19. @kit3bus Git LFS Filter Protocol: One Git LFS invocation Git Checkout Git
20. @kit3bus Taylor Code Design Decisions Lars Git Contribution Process Git Checkout Git LFS Git
21. @kit3bus How to Contribute to Git Core
22. @kit3bus Subscribe to the Mailing List Send an email to "majordomo@vger.kernel.org" with an empty subject line and the words "subscribe git" in the message body.
23. @kit3bus Search the Mailing List Archive http://public-inbox.org/git/
24. @kit3bus Write an RFC* Pro Tip:
 Measure 
 performance * Request for Comment
25. @kit3bus Mailing List Tips & Tricks Inline replies Text only
26. @kit3bus Make Commits $ git checkout -b myfeature master ... feature ... ... tests ... ... docs ... $ git commit maint for bugfixes Read Documentation/CodingGuidelines
27. @kit3bus Make Commits area: short description Explain the problem. Explain the solution. Signed-off-by: Lars Schneider <mail@xyz.com> Pro Tip:
 Use a spell checker!
28. @kit3bus Check Commits make git rebase master -x "make" make test git diff --check master...HEAD ... https://github.com/larsxschneider/git-list-helper prepare-patch.sh master Lazy?
29. @kit3bus Check Commits Pro Tip:
 Enable TravisCI
30. @kit3bus Prepare Patch Emails git format-patch master From ea25a1834be7e2da36fd00f4abc403a56d6b35bc Mon Sep 17 00:00:00 2001 From: Lars Schneider <larsxschneider@gmail.com> Date: Mon, 21 Nov 2016 13:44:08 +0100 Subject: [PATCH v1] short description Detailed commit message. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> --- Hi, message to mailing list! Cheers, Lars apply.c | 2 +- archive.c | 2 +- 2 files changed, 145 insertions(+), 24 deletions(-) diff --git a/apply.c b/apply.c index 2ed808d429..aa7e6e4359 100644 --- a/apply.c +++ b/apply.c @@ -4328,7 +4328,7 @@ static int try_create_file - if (convert_to_working_tree(path, buf, size, &nbuf)) { + if (convert_to_working_tree(path, buf, size, &nbuf, NULL)) { } -- 2.11.0 Commit Message Additional explanations/ cover letter Diff statistics Patch Your Git version Email header
31. @kit3bus Prepare Patch Emails prepare-patch.sh master From ea25a1834be7e2da36fd00f4abc403a56d6b35bc Mon Sep 17 00:00:00 2001 From: Lars Schneider <larsxschneider@gmail.com> Date: Mon, 21 Nov 2016 13:44:08 +0100 Subject: [PATCH v1] short description Detailed commit message. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> --- Notes: Base Commit: 787f75f056 (master) Diff on Web: https://github.com/larsxschneider/git/commit/a8ec423355 Checkout: git fetch https://... demo && git checkout a8ec423355 apply.c | 2 +- archive.c | 2 +- 2 files changed, 145 insertions(+), 24 deletions(-) diff --git a/apply.c b/apply.c index 2ed808d429..aa7e6e4359 100644 --- a/apply.c +++ b/apply.c @@ -4328,7 +4328,7 @@ static int try_create_file - if (convert_to_working_tree(path, buf, size, &nbuf)) { + if (convert_to_working_tree(path, buf, size, &nbuf, NULL)) { } -- 2.11.0 Additional explanations/ cover letter
32. @kit3bus git send-email /patches/* --to=git@vger.kernel.org git send-email /patches/* --to=git@vger.kernel.org --cc=re@view.er Send Patch Emails Pro Tip:
 Find reviewers
 with git blame prepare-patch.sh master Lazy?
33. @kit3bus Wait...
34. @kit3bus Respond to Reviews
35. @kit3bus Respond to Reviews 1. Find agreement 2. Update commits 3. Check commits 4. Prepare patch emails Pro Tip:
 Make notes https://github.com/larsxschneider/git-list-helper prepare-patch.sh master 2
36. @kit3bus Respond to Reviews 1. Find agreement 2. Update commits 3. Check commits 4. Prepare patch emails 5. Send patch emails and CC reviewers Pro Tip:
 Make notes
37. @kit3bus v1 Contribution Cycle Take in
 reviews Send patches RFC
38. @kit3bus v1 v2 ... Contribution Cycle Take in
 reviews Send patches RFC
39. @kit3bus v1 v2 ... v11 Contribution Cycle Take in
 reviews next master Release! Send patches RFC More: Documentation/SubmittingPatches
40. @kit3bus Filter Protocol Statistics 1,628 changes 11 iterations 380 emails 6 months Pro Tip:
 Don't give up!
41. 34 Git LFS Next steps
42. 38 October, 2016 December, 2016 November, 2016 filter process is merged into git/next Git v2.11 release
43. 39 (master) $ git checkout my-feature (my-feature) $ ...
44. 40 (master) $ GIT_TRACE_PACKET=1 git checkout my-feature packet: git> command=smudge packet: git> pathname=foo/bar.dat packet: git> 0000 packet: git> CONTENT packet: git> 0000 packet: git< status=success packet: git< 0000 packet: git< SMUDGED_CONTENT packet: git< 0000 packet: git< 0000 (my-feature) $ ...
45. 44 3 1 2
46. Start small. 45
47. Validate quickly. 46
48. Iterate later. 47
49. 49 Integration tests
51. 51 Integration tests pkt-line protocol
53. 53 Integration tests pkt-line protocol process filter protocol
55. 59 for scanner.Scan() { req := scan.Request() switch req.Header[“command”] { case “clean”: case “smudge”: default: } } if err := scan.Err(); err != nil { // ... }
56. 60 type Request struct { Header map[string]string Payload []byte }
57. 62 Integration tests pkt-line protocol process filter protocol pkt-line I/O pt. I
59. 64 type io.Reader interface { Read(p []byte) (n int, err error) }
60. 65 type Pktline struct { ... } func (p *Pktline) Pkt() ([]byte, error) { // ... }
61. 72 type PktlineReader struct { ... } func (r *PktlineReader) Read(p []byte) (n int, err error) { for { pkt, err := r.pkt() if err != nil { return n, err } copy(p[n:], pkt) } return n, nil }
62. 73 for scanner.Scan() { req := scanner.Request() sig := sha256.New() io.Copy(sig, req.Body) fmt.Fprintln(“OID: %x”, sig.Sum(nil)) }
63. 77 Integration tests pkt-line protocol process filter protocol pkt-line I/O pt. II pkt-line I/O pt. I
65. 79 type io.Writer interface { Write(p []byte) (n int, err error) }
66. 87 type PktlineWriter struct { ... } func (w *PktlineWriter) Write(p []byte) (n int, err error) { var n int for n < len(p) { c := min(len(p[n:]), MaxPacketLength) if err := w.writePkt(p[n:c]); err != nil { return n, err } n += c } return n, nil }
67. 92 func smudge(out io.Writer, oid string) (n int, err error) { path := lfs.LocalMediaPath(oid) f, err := os.Open(path) if err != nil { return 0, err } return io.Copy(out, f) }
68. 93 Integration tests pkt-line protocol process filter protocol pkt-line I/O pt. II pkt-line I/O pt. I
69. ! Release! 94
70. 80x faster. 95
71. Next steps? • Promise capability (discussion in git-core) • File-path exchange • ??? 100
72. Thank you. 101