Git at Scale

Slides used in my talk at Git Merge 2016 in New York.
http://git-merge.com/

Transcript

1. Adopt Git at Scale and Stay Sane Lars Schneider Technical Lead GitHub Solutions Autodesk larsxschneider kit3bus lars.schneider@autodesk.com
2. Adopting Git is a Challenge ... 4000 engineers
3. Today's topics? 1. Ease Git on-boarding 2. Git usage recommendations 3. Git usage monitoring
4. What protocol to use? HTTPS SSH Git No authentication Difficult on Windows
5. What protocol to use? HTTPS SSH Git Recommended
6. HTTPS protocol $ git clone https://autodesk.com/myorg/myrepo Cloning into 'myrepo'... Username for 'https://autodesk.com': "Why do I have to type my password all the time?"
7. HTTPS protocol $ git clone https://autodesk.com/myorg/myrepo Cloning into 'myrepo'... Username for 'https://autodesk.com': Git Credential Helper Solution "Why do I have to type my password all the time?"
8. User Config "Why are my commits not associated to me?"
9. User Config "Why are my commits not associated to me?" $ git config --global user.name "Lars Schneider" $ git config --global user.email lars.schneider@autodesk.com Setup user!
10. Insight Trivial tasks performed by 4000 people generate many support tickets.
11. Enterprise Config for Git Painless Git setup with an easy way to share Git configs and scripts within a company.
12. Enterprise Config for Git $ git [setup command]
13. Enterprise Config for Git $ git adsk
14. Enterprise Config for Git $ git adsk Setup credential helper Check Git / Git LFS versions Self update Local user setup
15. Enterprise Config for Git $ git adsk Setup credential helper Check Git / Git LFS versions Self update Local user setup Too lazy to type? Go here: git.io/vajMI $ printf "protocol=https\nhost=github.com\n\n" | 
 git credential-$(git config credential.helper) get

16. Enterprise Config for Git $ git adsk Setup credential helper Check Git / Git LFS versions Self update Local user setup $ printf "protocol=https\nhost=github.com\n\n" | 
 git credential-$(git config credential.helper) get
 password=SECRET123 username=larsxschneider Oh...
17. Enterprise Config for Git $ git adsk Setup credential helper with token Check Git / Git LFS versions Self update Local user setup
18. Enterprise Config for Git $ git adsk Check signed Source Code Agreement ... Setup credential helper with token Check Git / Git LFS versions Self update Local user setup
19. Enterprise Config for Git [alias] adsk = "!f() { \ GIT_KIT=$(dirname $(git config include.path)) && \ bash \"$GIT_KIT/setup.sh\" \ }; f" File: /Users/lars/.enterprise/config.include New Git command! [include] path = /Users/lars/.enterprise/config.include File:~/.gitconfig Engineers global
 Git config
20. Enterprise Config for Git $ git clone https://autodesk.com/enterprise.git ~/.enterprise $ git config --global include.path ~/.enterprise/config.include Install: ... works on Windows, too. However, we made a custom version of the great Git for Windows Installer.
21. Enterprise Config for Git github.com/autodesk/enterprise-config-for-git
22. Git Usage Recommendations
23. Git Usage Recommendations 1. Create branches
24. Git Usage Recommendations 1. Create branches 2. Push often
25. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase)
26. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!)
27. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io)
28. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance)
29. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo)
30. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
31. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
32. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $ git push Git push is scary in an enterprise environment.
33. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $ git push fatal: '/// ATTENTION /// Do you want to push to github.com? Git push is scary in an enterprise environment.
34. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $ git push fatal: '/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com/ larsxschneider/test.git' Git push is scary in an enterprise environment.
35. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $ git push fatal: '/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com/ larsxschneider/test.git' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Use git config URL rewrite to create a warning: [url "/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com"] pushInsteadOf = "https://github.com" File: /Users/lars/.enterprise/config.include 1 2
36. Git Push Protection SourceTree
37. Git Push Protection SmartGit
38. Git Push Protection Google Summer of Code / Outreachy Project More info: https://git.github.io/SoC-2016-Ideas/ http://outreachy.org https://summerofcode.withgoogle.com
39. Git Usage Recommendations 1. Create branches 2. Push often 3. Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
40. Git and Large Files Files that change often and are large after compression are bad for Git.
41. Git and Large Files Files that change often and are large after compression are bad for Git. # binaries x avg. changes/year x avg. size in MB < 100 1000 x 1 x 0.05 = 25 10 x 365 x 0.05 = 183
42. Git LFS - fast clone $ git clone <URL> ... Downloading file1.bin (10 MB) Downloading file2.bin (40 MB) Downloading file3.bin (30 MB) ... Downloading file500.bin (20 MB) Takes ages!
43. Git LFS - fast clone – Rick Olsen (@technoweenie) “Use GIT_LFS_SKIP_SMUDGE and git-lfs pull to speed up cloning.”
44. Git LFS - fast clone Still slow! 13x faster!
45. Git LFS - fast clone $ git clone <URL> ... Downloading file1.bin (10 MB) Downloading file2.bin (40 MB) Downloading file3.bin (30 MB) ... Downloading file500.bin (20 MB) Takes ages! $ git adsk clone <URL> ... Git LFS: (500 of 500 files) Lightning fast!
46. Git LFS - fast clone
47. Git LFS - fast clone Wow!
48. Git Usage Monitoring
49. Git Usage Monitoring Log forwarding
50. Git Usage Monitoring host="<your GitHub Enterprise IP>" source="udp:514" sourcetype=syslog \"program\":\"upload-pack\" \"cloning\":true | rex "^.*github_audit\[\d+\]\:\s(?P<json>.+)" | spath input=json | bucket_time span=1d | stats count BY user_id, repo_id, _time Who is cloning all the time? Log forwarding
51. Git Usage Monitoring host="<your GitHub Enterprise IP>" source="udp:514" sourcetype=syslog \"program\":\"upload-pack\" \"cloning\":true | rex "^.*github_audit\[\d+\]\:\s(?P<json>.+)" | spath input=json | bucket_time span=1d | stats count BY user_id, repo_id, _time Who is cloning all the time? Log forwarding
52. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup
53. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
54. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
55. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
56. Takeaways 1. Script your Git on-boarding 2. Credential helpers are not secure 3. Articulate Git usage recommendations 4. Monitor your Git usage
57. Enterprise Config for Git github.com/autodesk/enterprise-config-for-git
58. Adopt Git at Scale and Stay Sane Lars Schneider Technical Lead GitHub Solutions Autodesk larsxschneider kit3bus lars.schneider@autodesk.com