Git LFS - How to handle large Git repos?

Video: https://youtu.be/YQzNfb4IwEY

Git became really successful in the Open Source world and, more recently, in the corporate world as well. However, Git is not suited for large files by design which is a major obstacle for many adopters. GitLFS is an extension to Git which addresses this problem. In this talk I will explain how GitLFS works and how Autodesk uses it at scale for 4000+ engineers and 1000+ repositories. You will also learn how to use it properly and how to avoid the most common pitfalls. If you have worked with Git before then this talk will prepare you to handle large files without trouble in Git!

Transcript

1. GIT LFS - How to handle large Git repos? - Lars Schneider Technical Lead 
 for Git at Autodesk Git and Git LFS contributor @kit3bus
2. What is a large Git repo?
3. Large number of files https://github.com/Microsoft/GVFS
4. Large number of commits
5. Large files
6. Why are large files a problem? = 100 MB video file
7. Clone: Large files in Git master 301MB + 100 MB + 100 MB 1 MB 100 MB 1 MB 100 MB 1 MB 100 MB
8. 301 MB 
 x 
 Number of Engineers Clone: Large files in Git 1 10 1 10 1 10 ... Git Server
9. Large files in Git LFS Large File Storage
 https://git-lfs.github.com
10. LFS Server Clone: Large files in Git LFS 1MB master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB
11. LFS Server Large files in Git LFS master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB Clone: 1 MB + Checkout: 100 MB
12. LFS Server Large files in Git LFS master 1 MB 100 MB 1 MB 1 MB 100 MB 0 MB 0 MB 0 MB 100 MB Clone: 1 MB + Checkout: 100 MB 60% saving!
13. How to use Git LFS?
14. How to use Git LFS? tracking only based on file name, file size is not relevant! |- .git | |- code.cpp - readme.md - video.mp4 $ git lfs track "*.mp4"
15. How to use Git LFS? |- .git |- .gitattributes |- code.cpp - readme.md - video.mp4 $ git lfs track "*.mp4" $ git add .
16. How to use Git LFS? |- .git |- .gitattributes |- code.cpp - readme.md - video.mp4 $ git lfs track "*.mp4" $ git add . $ git commit
17. How to use Git LFS? |- .git |- .gitattributes |- code.cpp - readme.md - video.mp4 $ git lfs track "*.mp4" $ git add . $ git commit $ git push
18. How to use Git LFS? |- .git |- .gitattributes |- code.cpp - readme.md - video.mp4 $ git lfs track "*.mp4" $ git add . $ git commit $ git push ! That's it!
19. Autodesk uses Git LFS
20. Autodesk uses 
 Git LFS • Best known for AutoCAD
 2D and 3D computer-aided design
 • 35 years in business
 • 4000 engineers, hundreds of products, terabytes of code and asset data
21. • Integration Test Data
 (3D Models, ...)
 • Auxiliary Data
 (Documentation, Images, Videos, ...)
 • Build Artifacts
 (not recommended) Autodesk uses 
 Git LFS
22. What have we learned ? 22
23. Developer (INCLUDES DESIGNER, TESTER, ...)
24. What is too large for Git?
25. What files should go to LFS? Mon Tue Wed Files that 
 do not compress well and change frequently Don't worry about files smaller than 500kb
26. git lfs track "*.png" How to track Git LFS files? Use Git 2.11+ and Git LFS 1.5+ for 80x speed improvement! 
 http://autode.sk/git-filter
27. git lfs track "*.lfs.*" e.g. /images/mountain.lfs.png How to track Git LFS files?
28. git lfs track "/big/*" e.g. /big/mountain.png How to track Git LFS files?
29. git lfs track "/mountain.png" How to track Git LFS files?
30. How to track Git LFS files? git lfs track "*.png" Case sensitive: Win/macOS: demo.PNG Linux: demo.PNG
31. How to track Git LFS files? git lfs track "*.png" Case sensitive: git lfs track "*.[pP][nN][gG]" Case in-sensitive: Use "git lfs ls-files" to check what files are tracked! Win/macOS: demo.PNG Linux: demo.PNG
32. Git LFS Gotchas
33. Gotchas Speed up Clones git lfs clone <URL> I'm working on making 
 this wrapper obsolete!
34. Gotchas Setup your Git credential helper 
 (or use SSH)!
35. Gotchas No line ending conversions 
 on Git LFS files! FirstLine SecondLine FirstLine▯Second Line
 Windows macOS
36. Gotchas $ git <command> Your branch is up-to-date with 'origin/master'. Encountered 2 file(s) that should have been pointers, but weren't: path/to/some/file.PNG other/image.PNG
37. Administrator 38 MANAGING GIT REPOS
38. Setup Git LFS 
 on dev machines! Enterprise Config for Git
 https://autode.sk/git
39. Distribute recent Git / Git LFS versions
40. Configure a file size limit on your Git server!
41. Takeaways • Git LFS works at scale • Use smart tracking patterns • Speed up clones with `git lfs clone` • Reject large files on the Git server Questions? @kit3bus