exploring git repos with SQL gitbase and source{d} engine

Francesc Campoy VP of Developer Relations [email protected] @francesc github.com/campoy

just for func youtube.com/justforfunc

just for func twitch.tv/justforfunclive li !

Agenda ● What is gitbase? ● How was it built? ● Q&A

What is gitbase?

github.com/src-d/gitbase ● SQL interface to git repositories ● Open Source ● Written in Go

LANGUAGE(path, content): Returns the language of a file given its path and contents. Powered by github.com/src-d/enry. Some custom functions

Lines of code per language # total lines of code per language in the Go repo SELECT lang, SUM(lines) as total_lines FROM ( SELECT t.tree_entry_name as name, LANGUAGE(t.tree_entry_name, b.blob_content) AS lang, ARRAY_LENGTH(SPLIT(b.blob_content, '\n')) as lines FROM refs r NATURAL JOIN commits c NATURAL JOIN commit_trees ct NATURAL JOIN tree_entries t NATURAL JOIN blobs b WHERE r.ref_name = 'HEAD' ) AS lines WHERE lang is not null GROUP BY lang ORDER BY total_lines DESC;

Lines of code per language # total lines of code per language in the Go repo SELECT lang, SUM(lines) as total_lines FROM ( SELECT t.tree_entry_name as name, LANGUAGE(t.tree_entry_name, b.blob_content) AS lang, ARRAY_LENGTH(SPLIT(b.blob_content, '\n')) as lines FROM refs r NATURAL JOIN commits c NATURAL JOIN commit_trees ct NATURAL JOIN tree_entries t NATURAL JOIN blobs b WHERE r.ref_name = 'HEAD' ) AS lines WHERE lang is not null GROUP BY lang ORDER BY total_lines DESC;

Some custom functions UAST(content, language, [filter]): Returns the Universal Abstract Syntax Tree resulting of parsing the given content in the given language. Powered by github.com/bblfsh/bblfshd.

SELECT files.repository_id, files.file_path, ARRAY_LENGTH(UAST( files.blob_content, LANGUAGE(files.file_path, files.blob_content), '//*[@roleFunction and @roleDeclaration]') ) as functions FROM files NATURAL JOIN refs WHERE LANGUAGE(files.file_path,files.blob_content) = 'Go' AND refs.ref_name = 'HEAD' Number of functions per Go file

SELECT files.repository_id, files.file_path, ARRAY_LENGTH(UAST( files.blob_content, LANGUAGE(files.file_path, files.blob_content), '//*[@roleFunction and @roleDeclaration]') ) as functions FROM files NATURAL JOIN refs WHERE LANGUAGE(files.file_path,files.blob_content) = 'Go' AND refs.ref_name = 'HEAD' Number of functions per Go file

source{d} Engine ● Too many moving pieces ● Too many steps to get started ● Solving it all with the power of Docker!

No content

https://medium.com/sourcedtech

Why?

No content

Demo Time!

How was gitbase built?

vitess github.com/vitessio/vitess (by YouTube)

youtu.be/midJ6b1LkA0

go-mysql-server github.com/src-d/go-mysql-server ● Ready to run MySQL server ● Extensible via interfaces Database and Table ● Example: github.com/campoy/csvql

Making it go faster

Indexes ● SQL Indexes can speed up queries substantially ● Vitess doesn’t provide this ● Pilosa does!

Caches ● Caching is the obvious option to make queries faster ● We didn’t want to reinvent the wheel ● We didn’t have to, thanks to Hashicorp ● Based on github.com/golang/groupcache

● The regexp package in Go is linear, but not always faster https://swtch.com/~rsc/regexp/regexp1.html ● Alternative: github.com/moovweb/rubex (onigurama) Regular Expressions

We want you!

And we’re hiring!

Thanks! @francesc