Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Git infrastructure with Go
Search
David Calavera
February 21, 2015
Programming
3
570
Git infrastructure with Go
Talk for Gophercon India 2015 about using Go to manage large git data system.
David Calavera
February 21, 2015
Tweet
Share
More Decks by David Calavera
See All by David Calavera
The language of Chatops
calavera
0
140
How GitHub develops, ships and supports GitHub Enterprise
calavera
2
460
Project Warbird
calavera
1
160
Trinidad
calavera
0
32
JRuby hacking guide
calavera
0
57
My name is Trinidad
calavera
0
54
Other Decks in Programming
See All in Programming
株式会社 Sun terras カンパニーデック
sunterras
0
330
AI Agent 時代的開發者生存指南
eddie
3
1.8k
(Extension DC 2025) Actor境界を越える技術
teamhimeh
1
260
開発生産性を上げるための生成AI活用術
starfish719
3
1.2k
Goで実践するドメイン駆動開発 AIと歩み始めた新規プロダクト開発の現在地
imkaoru
4
850
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
ioquatix
4
2.2k
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
3.4k
テーブル定義書の構造化抽出して、生成AIでDWH分析を試してみた / devio2025tokyo
kasacchiful
0
110
チームの境界をブチ抜いていけ
tokai235
0
180
Introduce Hono CLI
yusukebe
6
2.6k
エンジニアインターン「Treasure」とHonoの2年、そして未来へ / Our Journey with Hono Two Years at Treasure and Beyond
carta_engineering
0
310
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
410
Featured
See All Featured
Building Adaptive Systems
keathley
44
2.8k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
RailsConf 2023
tenderlove
30
1.2k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
A better future with KSS
kneath
239
18k
It's Worth the Effort
3n
187
28k
Making the Leap to Tech Lead
cromwellryan
135
9.6k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
9
910
Why Our Code Smells
bkeepers
PRO
340
57k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
Code Review Best Practice
trishagee
72
19k
Transcript
High performance git infrastructure with Go David Calavera Code Climate
None
2005
(╯°□°)╯︵ ┻━┻
2008
2010…
How do we make git faster?
* system(“git log …”) * Ruby for brevity
git fetch origin
None
github.com/libgit2/git2go Disclaimer: examples do not handle errors
// Load with side effects. // Initialize libgit2’s TLS: //
// func init() { // C.git_libgit2_init() // } // // Import package “git”, which is // not very goimports friendly. import “github.com/libgit2/git2go”
// Create a new repository. // Do not use a
working directory. path := “/var/git/repository” bare := true r, _ := git.InitRepository(path, bare) ! // Clone a repository. url := “git://github.com/golang/go” opts := git.CloneOptions{Bare: bare} r, _ := git.Clone(url, path, &opts)
// Create a new remote ref. name := “my-fork” url
:= “git://github.com/wadus/go” rm, _ := r.CreateRemote(name, url) ! // Fetch all refs from a remote. var refspecs []string rm.Fetch(refspecs, nil, nil)
// Search for objects. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
! c, _ := r.LookupCommit(oid) b, _ := r.LookupBlob(oid) t, _ := r.LookupTree(oid) ! o, _ := r.Lookup(oid) fmt.Printf(“ %v\n”, o.Type())
// Read commit data. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
path := “src/os/exec.go” ! c, _ := r.LookupCommit(oid) t, _ := c.Tree() e, _ := t.EntryByPath(path) ! b, _ := r.LookupBlob(e.Id()) fmt.Printf(“ %q\n”, b.Contents())
// Commit new changes. idx, _ := r.Index() idx.AddByPath(“src/os/exec.go”) t,
_ := idx.WriteTree() idx.Write() ! h, _ := r.Head() c, _ := r.LookupCommit(h) ! s := &git.Signature{“me”, “
[email protected]
”, time.Now()} m := “Add moar changes” r.CreateCommit(“”, s, s, m, t, c)
None
None
Designing a distributed git storage
Application Git Service
Constraint your data model
// protocol buffers schema. message Branch { required string name
= 1; } ! message Repository { optional string name = 1; repeated Branch branches = 2; }
// Read branches. var branches []*pb.Branch ! f := func(b
*git.Branch, t git.BranchType) error { n, _ := b.Name() p := &pb.Branch{ Name: &n, } branches = append(branches, p) return nil } ! b, _ := r.NewBranchIterator(git.BranchRemote) b.ForEach(f)
// Read branches via http. h := func(w http.ResponseWriter, r
*http.Request) { pbBranches := readBranches(r) pbRepo := &pb.Repository{ Branches: pbBranches, } ! data, _ := proto.Marshal(pbRepo) w.Write(data) } ! http.HandleFunc(“/r/foo/branches”, h)
Design from first principles
A shared-data system can have at most two of the
three following properties: Consistency, Availability, and tolerance to network Partitions Dr. Eric Brewer
You Can’t Sacrifice Partition Tolerance Coda Hale
github.com/afex/hystrix-go" github.com/rubyist/circuitbreaker" github.com/eapache/go-resiliency/braker
// Read branches via http. import “github.com/rubyist/circuitbreaker” ! out :=
5 * time.Second url := “http://git-server/r/foo/branches” c := circuit.NewHTTPClient(out, 10, nil) ! c.BreakerTripped = func() { // Handle partition error response. } ! resp, _ := c.Get(url)
Replication Consistency Vs Availability
Application Git Service Git Service Primary Replica
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Done!
Application Git Service Git Service Primary Replica Done!
// Handle fetch requests. h := func(w http.ResponseWriter, r *http.Request)
{ peerChannel := replicateRequest(r) rm, _ := repo.LoadRemote(“origin”) var refspecs []string rm.Fetch(refspecs, nil, nil) ! if peersChannel != nil { waitForPeers(peerChannel) } w.WriteHeader(201) } ! http.HandleFunc(“/r/foo/fetch”, h)
// Replicate request. func replicateRequest(r *http.Request) chan int { if
req.Header.Get(“X-GIT-REPLICATE”) != “” { return nil } peerChannel := make(chan int) replicaURL, err := url.Parse(replicaHost) replicaURL.Path = r.Path replicaURL.Header.Set(“X-GIT-REPLICATE”, “true") req, _ := http.NewRequest(“POST”, replicaURL.String(), nil) ! go func() { resp, _ := httpClient.Do(req) peerChannel <- resp.StatusCode }() return peerChannel }
// Wait for replica response. func waitForPeers(channel chan int) error
{ replicaStatus := <- channel switch replicaStatus { case 201: // default: // ☔️ ☔️ } ! return nil }
git architectures are fun
Thank you! @calavera