Git at Scale

Git at Scale

Slides used in my talk at Git Merge 2016 in New York.
http://git-merge.com/

A930e33e0dfa43bafa0f8b0b05dba8ca?s=128

Lars Schneider

April 05, 2016
Tweet

Transcript

  1. Adopt Git at Scale and Stay Sane Lars Schneider Technical

    Lead GitHub Solutions Autodesk larsxschneider kit3bus lars.schneider@autodesk.com
  2. Adopting Git is a Challenge ... 4000 engineers

  3. Today's topics? 1. Ease Git on-boarding 2. Git usage recommendations

    3. Git usage monitoring
  4. What protocol to use? HTTPS SSH Git No authentication Difficult

    on Windows
  5. What protocol to use? HTTPS SSH Git Recommended

  6. HTTPS protocol $ git clone https://autodesk.com/myorg/myrepo Cloning into 'myrepo'... Username

    for 'https://autodesk.com': "Why do I have to type my password all the time?"
  7. HTTPS protocol $ git clone https://autodesk.com/myorg/myrepo Cloning into 'myrepo'... Username

    for 'https://autodesk.com': Git Credential Helper Solution "Why do I have to type my password all the time?"
  8. User Config "Why are my commits not associated to me?"

  9. User Config "Why are my commits not associated to me?"

    $ git config --global user.name "Lars Schneider" $ git config --global user.email lars.schneider@autodesk.com Setup user!
  10. Insight Trivial tasks performed by 4000 people generate many support

    tickets.
  11. Enterprise Config for Git Painless Git setup with an easy

    way to share Git configs and scripts within a company.
  12. Enterprise Config for Git $ git [setup command]

  13. Enterprise Config for Git $ git adsk

  14. Enterprise Config for Git $ git adsk Setup credential helper

    Check Git / Git LFS versions Self update Local user setup
  15. Enterprise Config for Git $ git adsk Setup credential helper

    Check Git / Git LFS versions Self update Local user setup Too lazy to type? Go here: git.io/vajMI $ printf "protocol=https\nhost=github.com\n\n" | 
 git credential-$(git config credential.helper) get

  16. Enterprise Config for Git $ git adsk Setup credential helper

    Check Git / Git LFS versions Self update Local user setup $ printf "protocol=https\nhost=github.com\n\n" | 
 git credential-$(git config credential.helper) get
 password=SECRET123 username=larsxschneider Oh...
  17. Enterprise Config for Git $ git adsk Setup credential helper

    with token Check Git / Git LFS versions Self update Local user setup
  18. Enterprise Config for Git $ git adsk Check signed Source

    Code Agreement ... Setup credential helper with token Check Git / Git LFS versions Self update Local user setup
  19. Enterprise Config for Git [alias] adsk = "!f() { \

    GIT_KIT=$(dirname $(git config include.path)) && \ bash \"$GIT_KIT/setup.sh\" \ }; f" File: /Users/lars/.enterprise/config.include New Git command! [include] path = /Users/lars/.enterprise/config.include File:~/.gitconfig Engineers global
 Git config
  20. Enterprise Config for Git $ git clone https://autodesk.com/enterprise.git ~/.enterprise $

    git config --global include.path ~/.enterprise/config.include Install: ... works on Windows, too. However, we made a custom version of the great Git for Windows Installer.
  21. Enterprise Config for Git github.com/autodesk/enterprise-config-for-git

  22. Git Usage Recommendations

  23. Git Usage Recommendations 1. Create branches

  24. Git Usage Recommendations 1. Create branches 2. Push often

  25. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase)
  26. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!)
  27. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io)
  28. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance)
  29. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo)
  30. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
  31. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
  32. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $

    git push Git push is scary in an enterprise environment.
  33. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $

    git push fatal: '/// ATTENTION /// Do you want to push to github.com? Git push is scary in an enterprise environment.
  34. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $

    git push fatal: '/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com/ larsxschneider/test.git' Git push is scary in an enterprise environment.
  35. Git Push Protection $ git clone https://github.com/larsxschneider/test.git $ ... $

    git push fatal: '/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com/ larsxschneider/test.git' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Use git config URL rewrite to create a warning: [url "/// ATTENTION /// Do you want to push to github.com? Disable this warning with: 'git adsk enable-push github.com"] pushInsteadOf = "https://github.com" File: /Users/lars/.enterprise/config.include 1 2
  36. Git Push Protection SourceTree

  37. Git Push Protection SmartGit

  38. Git Push Protection Google Summer of Code / Outreachy Project

    More info: https://git.github.io/SoC-2016-Ideas/ http://outreachy.org https://summerofcode.withgoogle.com
  39. Git Usage Recommendations 1. Create branches 2. Push often 3.

    Merge instead of rebase (at least until you grok rebase) 4. Avoid cherry picking (a commit is no delta!) 5. Configure gitignore (use gitignore.io) 6. Avoid many files (> 100k hurts performance) 7. Use Submodules (use automation; avoid nesting; avoid >25 per repo) 8. Use Git-LFS for large files
  40. Git and Large Files Files that change often and are

    large after compression are bad for Git.
  41. Git and Large Files Files that change often and are

    large after compression are bad for Git. # binaries x avg. changes/year x avg. size in MB < 100 1000 x 1 x 0.05 = 25 10 x 365 x 0.05 = 183
  42. Git LFS - fast clone $ git clone <URL> ...

    Downloading file1.bin (10 MB) Downloading file2.bin (40 MB) Downloading file3.bin (30 MB) ... Downloading file500.bin (20 MB) Takes ages!
  43. Git LFS - fast clone – Rick Olsen (@technoweenie) “Use

    GIT_LFS_SKIP_SMUDGE and git-lfs pull to speed up cloning.”
  44. Git LFS - fast clone Still slow! 13x faster!

  45. Git LFS - fast clone $ git clone <URL> ...

    Downloading file1.bin (10 MB) Downloading file2.bin (40 MB) Downloading file3.bin (30 MB) ... Downloading file500.bin (20 MB) Takes ages! $ git adsk clone <URL> ... Git LFS: (500 of 500 files) Lightning fast!
  46. Git LFS - fast clone

  47. Git LFS - fast clone Wow!

  48. Git Usage Monitoring

  49. Git Usage Monitoring Log forwarding

  50. Git Usage Monitoring host="<your GitHub Enterprise IP>" source="udp:514" sourcetype=syslog \"program\":\"upload-pack\"

    \"cloning\":true | rex "^.*github_audit\[\d+\]\:\s(?P<json>.+)" | spath input=json | bucket_time span=1d | stats count BY user_id, repo_id, _time Who is cloning all the time? Log forwarding
  51. Git Usage Monitoring host="<your GitHub Enterprise IP>" source="udp:514" sourcetype=syslog \"program\":\"upload-pack\"

    \"cloning\":true | rex "^.*github_audit\[\d+\]\:\s(?P<json>.+)" | spath input=json | bucket_time span=1d | stats count BY user_id, repo_id, _time Who is cloning all the time? Log forwarding
  52. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup

  53. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged

    Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
  54. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged

    Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
  55. Git Usage Monitoring /opt/backup-utils/bin/ghe-backup SELECT DATE_FORMAT(pull_requests.merged_at, '%Y-%m') AS date, 'Merged

    Pull Requests' AS type, COUNT(*) AS count FROM pull_requests AS pr, users WHERE pr.merged_at > (NOW() - INTERVAL 2 YEAR) AND pr.merged_at < DATE_FORMAT(CURRENT_DATE,'%Y-%m-01') AND user_id = users.id GROUP BY 1 ORDER BY 1,2 Merged Pull Requests per month?
  56. Takeaways 1. Script your Git on-boarding 2. Credential helpers are

    not secure 3. Articulate Git usage recommendations 4. Monitor your Git usage
  57. Enterprise Config for Git github.com/autodesk/enterprise-config-for-git

  58. Adopt Git at Scale and Stay Sane Lars Schneider Technical

    Lead GitHub Solutions Autodesk larsxschneider kit3bus lars.schneider@autodesk.com