Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Git infrastructure with Go
Search
David Calavera
February 21, 2015
Programming
3
570
Git infrastructure with Go
Talk for Gophercon India 2015 about using Go to manage large git data system.
David Calavera
February 21, 2015
Tweet
Share
More Decks by David Calavera
See All by David Calavera
The language of Chatops
calavera
0
140
How GitHub develops, ships and supports GitHub Enterprise
calavera
2
450
Project Warbird
calavera
1
160
Trinidad
calavera
0
32
JRuby hacking guide
calavera
0
57
My name is Trinidad
calavera
0
52
Other Decks in Programming
See All in Programming
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
230
AIと”コードの評価関数”を共有する / Share the "code evaluation function" with AI
euglena1215
1
140
WebViewの現在地 - SwiftUI時代のWebKit - / The Current State Of WebView
marcy731
0
110
童醫院敏捷轉型的實踐經驗
cclai999
0
210
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
1.9k
既存デザインを変更せずにタップ領域を広げる方法
tahia910
1
280
RailsGirls IZUMO スポンサーLT
16bitidol
0
170
Team operations that are not burdened by SRE
kazatohiei
1
310
Discover Metal 4
rei315
2
120
C++20 射影変換
faithandbrave
0
570
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
140
5つのアンチパターンから学ぶLT設計
narihara
1
160
Featured
See All Featured
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Building an army of robots
kneath
306
45k
Bash Introduction
62gerente
614
210k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
500
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Adopting Sorbet at Scale
ufuk
77
9.4k
Rebuilding a faster, lazier Slack
samanthasiow
82
9.1k
Transcript
High performance git infrastructure with Go David Calavera Code Climate
None
2005
(╯°□°)╯︵ ┻━┻
2008
2010…
How do we make git faster?
* system(“git log …”) * Ruby for brevity
git fetch origin
None
github.com/libgit2/git2go Disclaimer: examples do not handle errors
// Load with side effects. // Initialize libgit2’s TLS: //
// func init() { // C.git_libgit2_init() // } // // Import package “git”, which is // not very goimports friendly. import “github.com/libgit2/git2go”
// Create a new repository. // Do not use a
working directory. path := “/var/git/repository” bare := true r, _ := git.InitRepository(path, bare) ! // Clone a repository. url := “git://github.com/golang/go” opts := git.CloneOptions{Bare: bare} r, _ := git.Clone(url, path, &opts)
// Create a new remote ref. name := “my-fork” url
:= “git://github.com/wadus/go” rm, _ := r.CreateRemote(name, url) ! // Fetch all refs from a remote. var refspecs []string rm.Fetch(refspecs, nil, nil)
// Search for objects. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
! c, _ := r.LookupCommit(oid) b, _ := r.LookupBlob(oid) t, _ := r.LookupTree(oid) ! o, _ := r.Lookup(oid) fmt.Printf(“ %v\n”, o.Type())
// Read commit data. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
path := “src/os/exec.go” ! c, _ := r.LookupCommit(oid) t, _ := c.Tree() e, _ := t.EntryByPath(path) ! b, _ := r.LookupBlob(e.Id()) fmt.Printf(“ %q\n”, b.Contents())
// Commit new changes. idx, _ := r.Index() idx.AddByPath(“src/os/exec.go”) t,
_ := idx.WriteTree() idx.Write() ! h, _ := r.Head() c, _ := r.LookupCommit(h) ! s := &git.Signature{“me”, “
[email protected]
”, time.Now()} m := “Add moar changes” r.CreateCommit(“”, s, s, m, t, c)
None
None
Designing a distributed git storage
Application Git Service
Constraint your data model
// protocol buffers schema. message Branch { required string name
= 1; } ! message Repository { optional string name = 1; repeated Branch branches = 2; }
// Read branches. var branches []*pb.Branch ! f := func(b
*git.Branch, t git.BranchType) error { n, _ := b.Name() p := &pb.Branch{ Name: &n, } branches = append(branches, p) return nil } ! b, _ := r.NewBranchIterator(git.BranchRemote) b.ForEach(f)
// Read branches via http. h := func(w http.ResponseWriter, r
*http.Request) { pbBranches := readBranches(r) pbRepo := &pb.Repository{ Branches: pbBranches, } ! data, _ := proto.Marshal(pbRepo) w.Write(data) } ! http.HandleFunc(“/r/foo/branches”, h)
Design from first principles
A shared-data system can have at most two of the
three following properties: Consistency, Availability, and tolerance to network Partitions Dr. Eric Brewer
You Can’t Sacrifice Partition Tolerance Coda Hale
github.com/afex/hystrix-go" github.com/rubyist/circuitbreaker" github.com/eapache/go-resiliency/braker
// Read branches via http. import “github.com/rubyist/circuitbreaker” ! out :=
5 * time.Second url := “http://git-server/r/foo/branches” c := circuit.NewHTTPClient(out, 10, nil) ! c.BreakerTripped = func() { // Handle partition error response. } ! resp, _ := c.Get(url)
Replication Consistency Vs Availability
Application Git Service Git Service Primary Replica
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Done!
Application Git Service Git Service Primary Replica Done!
// Handle fetch requests. h := func(w http.ResponseWriter, r *http.Request)
{ peerChannel := replicateRequest(r) rm, _ := repo.LoadRemote(“origin”) var refspecs []string rm.Fetch(refspecs, nil, nil) ! if peersChannel != nil { waitForPeers(peerChannel) } w.WriteHeader(201) } ! http.HandleFunc(“/r/foo/fetch”, h)
// Replicate request. func replicateRequest(r *http.Request) chan int { if
req.Header.Get(“X-GIT-REPLICATE”) != “” { return nil } peerChannel := make(chan int) replicaURL, err := url.Parse(replicaHost) replicaURL.Path = r.Path replicaURL.Header.Set(“X-GIT-REPLICATE”, “true") req, _ := http.NewRequest(“POST”, replicaURL.String(), nil) ! go func() { resp, _ := httpClient.Do(req) peerChannel <- resp.StatusCode }() return peerChannel }
// Wait for replica response. func waitForPeers(channel chan int) error
{ replicaStatus := <- channel switch replicaStatus { case 201: // default: // ☔️ ☔️ } ! return nil }
git architectures are fun
Thank you! @calavera