Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Git infrastructure with Go
Search
David Calavera
February 21, 2015
Programming
3
570
Git infrastructure with Go
Talk for Gophercon India 2015 about using Go to manage large git data system.
David Calavera
February 21, 2015
Tweet
Share
More Decks by David Calavera
See All by David Calavera
The language of Chatops
calavera
0
140
How GitHub develops, ships and supports GitHub Enterprise
calavera
2
450
Project Warbird
calavera
1
160
Trinidad
calavera
0
32
JRuby hacking guide
calavera
0
57
My name is Trinidad
calavera
0
52
Other Decks in Programming
See All in Programming
生成AI時代のコンポーネントライブラリの作り方
touyou
1
290
AI Agent 時代のソフトウェア開発を支える AWS Cloud Development Kit (CDK)
konokenj
6
800
商品比較サービス「マイベスト」における パーソナライズレコメンドの第一歩
ucchiii43
0
180
リバースエンジニアリング新時代へ! GhidraとClaude DesktopをMCPで繋ぐ/findy202507
tkmru
3
960
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
680
AI駆動のマルチエージェントによる業務フロー自動化の設計と実践
h_okkah
0
230
MCPを使ってイベントソーシングのAIコーディングを効率化する / Streamlining Event Sourcing AI Coding with MCP
tomohisa
0
170
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
300
効率的な開発手段として VRTを活用する
ishkawa
0
160
ISUCON研修おかわり会 講義スライド
arfes0e2b3c
1
470
PipeCDのプラグイン化で目指すところ
warashi
1
300
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
200
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Designing for Performance
lara
610
69k
Documentation Writing (for coders)
carmenintech
72
4.9k
Writing Fast Ruby
sferik
628
62k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
750
Become a Pro
speakerdeck
PRO
29
5.4k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Gamification - CAS2011
davidbonilla
81
5.4k
BBQ
matthewcrist
89
9.7k
Faster Mobile Websites
deanohume
308
31k
Transcript
High performance git infrastructure with Go David Calavera Code Climate
None
2005
(╯°□°)╯︵ ┻━┻
2008
2010…
How do we make git faster?
* system(“git log …”) * Ruby for brevity
git fetch origin
None
github.com/libgit2/git2go Disclaimer: examples do not handle errors
// Load with side effects. // Initialize libgit2’s TLS: //
// func init() { // C.git_libgit2_init() // } // // Import package “git”, which is // not very goimports friendly. import “github.com/libgit2/git2go”
// Create a new repository. // Do not use a
working directory. path := “/var/git/repository” bare := true r, _ := git.InitRepository(path, bare) ! // Clone a repository. url := “git://github.com/golang/go” opts := git.CloneOptions{Bare: bare} r, _ := git.Clone(url, path, &opts)
// Create a new remote ref. name := “my-fork” url
:= “git://github.com/wadus/go” rm, _ := r.CreateRemote(name, url) ! // Fetch all refs from a remote. var refspecs []string rm.Fetch(refspecs, nil, nil)
// Search for objects. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
! c, _ := r.LookupCommit(oid) b, _ := r.LookupBlob(oid) t, _ := r.LookupTree(oid) ! o, _ := r.Lookup(oid) fmt.Printf(“ %v\n”, o.Type())
// Read commit data. sha := “4c279186e24f7b3a59aa682a870747df6eaca013” oid := git.NewOid(sha)
path := “src/os/exec.go” ! c, _ := r.LookupCommit(oid) t, _ := c.Tree() e, _ := t.EntryByPath(path) ! b, _ := r.LookupBlob(e.Id()) fmt.Printf(“ %q\n”, b.Contents())
// Commit new changes. idx, _ := r.Index() idx.AddByPath(“src/os/exec.go”) t,
_ := idx.WriteTree() idx.Write() ! h, _ := r.Head() c, _ := r.LookupCommit(h) ! s := &git.Signature{“me”, “
[email protected]
”, time.Now()} m := “Add moar changes” r.CreateCommit(“”, s, s, m, t, c)
None
None
Designing a distributed git storage
Application Git Service
Constraint your data model
// protocol buffers schema. message Branch { required string name
= 1; } ! message Repository { optional string name = 1; repeated Branch branches = 2; }
// Read branches. var branches []*pb.Branch ! f := func(b
*git.Branch, t git.BranchType) error { n, _ := b.Name() p := &pb.Branch{ Name: &n, } branches = append(branches, p) return nil } ! b, _ := r.NewBranchIterator(git.BranchRemote) b.ForEach(f)
// Read branches via http. h := func(w http.ResponseWriter, r
*http.Request) { pbBranches := readBranches(r) pbRepo := &pb.Repository{ Branches: pbBranches, } ! data, _ := proto.Marshal(pbRepo) w.Write(data) } ! http.HandleFunc(“/r/foo/branches”, h)
Design from first principles
A shared-data system can have at most two of the
three following properties: Consistency, Availability, and tolerance to network Partitions Dr. Eric Brewer
You Can’t Sacrifice Partition Tolerance Coda Hale
github.com/afex/hystrix-go" github.com/rubyist/circuitbreaker" github.com/eapache/go-resiliency/braker
// Read branches via http. import “github.com/rubyist/circuitbreaker” ! out :=
5 * time.Second url := “http://git-server/r/foo/branches” c := circuit.NewHTTPClient(out, 10, nil) ! c.BreakerTripped = func() { // Handle partition error response. } ! resp, _ := c.Get(url)
Replication Consistency Vs Availability
Application Git Service Git Service Primary Replica
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Fetch!
Application Git Service Git Service Primary Replica Done!
Application Git Service Git Service Primary Replica Done!
// Handle fetch requests. h := func(w http.ResponseWriter, r *http.Request)
{ peerChannel := replicateRequest(r) rm, _ := repo.LoadRemote(“origin”) var refspecs []string rm.Fetch(refspecs, nil, nil) ! if peersChannel != nil { waitForPeers(peerChannel) } w.WriteHeader(201) } ! http.HandleFunc(“/r/foo/fetch”, h)
// Replicate request. func replicateRequest(r *http.Request) chan int { if
req.Header.Get(“X-GIT-REPLICATE”) != “” { return nil } peerChannel := make(chan int) replicaURL, err := url.Parse(replicaHost) replicaURL.Path = r.Path replicaURL.Header.Set(“X-GIT-REPLICATE”, “true") req, _ := http.NewRequest(“POST”, replicaURL.String(), nil) ! go func() { resp, _ := httpClient.Do(req) peerChannel <- resp.StatusCode }() return peerChannel }
// Wait for replica response. func waitForPeers(channel chan int) error
{ replicaStatus := <- channel switch replicaStatus { case 201: // default: // ☔️ ☔️ } ! return nil }
git architectures are fun
Thank you! @calavera