CSI: Gopher

If we could intelligently parse all of the open-source Go code on GitHub, what could we learn? We’re going to show you some of the interesting things we’ve found in Go projects, from library usage, idioms & package layouts, to how Gophers can use this data to make decisions about their own APIs.

Transcript

1. CSI: Gopher
2. Francesc Campoy @francesc - funemployed Matt Silverlock @elithrar - Google
3. Two years ago, Francesc wrote a blog post
4. What are the most imported packages? Got some cool results
5. It wasn’t easy ...
9. Last year I attended a GoSF meetup
10. Francesc attending Matt’s talk
11. • Wrote a LOT of regex • Found some neat bugs! • Francesc had been working with tools that made what I was doing more accessible.
12. How do you find bugs or API (mis) usage? • Across microservice code-bases • In all consumers of your library? • Accurately?
13. Previously: ugly regular expressions
15. And: really ugly regular expressions
16. Obviously, this is No Good ™ How can we do better?
17. What's this talk actually about?
18. Your task Find the most common string literal in your source code
19. What’s a string? It depends: 'single quotes' "double quotes" '''triple single quotes''' """triple double quotes""" `back quotes` «Please» „make it“ qq§ STOP! §
20. https://xkcd.com/1171/
21. Oh, also prepare to escape your escaping characters ...
22. https://xkcd.com/1638/
23. Well, that’s a complicated* regular expression *impossible without recursive regular expressions recursive regular expressions?
24. Recursive regular expressions?
25. Well, we can parse the repositories as AST (Abstract Syntax Trees).
26. Universal Abstract Syntax Trees package main import "fmt" func main() { fmt.Println("Hello, gophers") } File Declaration Import “fmt” Declaration FunctionGroup main Body go:CallExpr fmt Println “Hello, gophers” Fun Args
27. Universal Abstract Syntax Trees print('''Hello, Pythonistas''') File CallExpr print “Hello, Pythonistas” Fun Args
28. A single tree format for all languages: a single tool for all
29. A single label set: a single concept list
30. Universal Abstract Syntax Trees uast:String XPath: //uast:String File Declaration Import “fmt” Declaration FunctionGroup main Body go:CallExpr fmt Println “Hello, gophers” Fun Args “fmt” “Hello, gophers”
31. “Hello, Pythonistas” Universal Abstract Syntax Trees '''Hello, Pythonistas''' uast:String XPath: //uast:String File CallExpr print Fun Args
32. Let’s run it!
33. While this is running ...
34. So, how are ASTs relevant here? - We downloaded ~19GB of Go repositories from GitHub* - Parsed them as UAST with src-d/engine - Queried the generated databases with SQL + UAST extensions https://github.com/src-d/engine
35. OK, so: what are some of the things we can investigate? - Finding ‘bad’ code - Best practices and idioms - Usage analysis of APIs
36. Investigation #1: Finding 'bad' code
37. "Bad crypto" - One of my favorite topics - How is a non-expert supposed to know that math/rand is bad vs. crypto/rand? - What about hash functions vs. KDFs?
40. Investigation #2: Best practices & idioms
45. Computer says ...
47. Number of init functions per file: mean 0.12
49. Number of init functions per file (0 to 10)
50. Number of init functions per file (log scale) 352 init functions LOL
51. Smells like vendoring ...
52. github.com/vmware/govmomi/vim25/types/enum.go
54. Applied the scientific method to best practices Make an observation. Ask a question. Form a hypothesis, or testable explanation. Make a prediction based on the hypothesis. Test the prediction. Iterate: use the results to make new hypotheses or predictions.
55. Investigation #3: API usage & breaking them
56. Premise: You maintain a popular OSS library. Problem: You're thinking about deprecating a method, but want to quantify the impact. How do you accurately find these cases?
57. Actual problem: • gorilla/context predates net/http's Request.Context() implementation. • Using both causes a memory leak, due to "islanding" the pointer to the original *Request. • Can we find these users & help?!
58. The Problem
59. Users!
60. Compared to...
61. What else?
62. • Order your godoc by most-used identifiers - rather than a bunch of ErrSomething at the top • Identify dependency version usage across your org, or as a maintainer. • Smart auto-completion based on previous usages of the API.
65. Tools we used and where to find them
66. Tools: • Google BigQuery • source{d} engine: github.com/src-d/engine • A lot of RAM, CPUs, and time Resources: • Finding Bugs with BigQuery & GitHub: bit.ly/gosf-bq • Analyzing Go code with BigQuery: bit.ly/go-bq
67. So … what about those strings?
69. #1 with 1,260,374 occurrences: '' #2 with 456,480 occurrences: 'fmt' #3 with 337,064 occurrences: 'json:"-"' Most common strings in Go
70. Other languages most common strings in Python: 'automanaged' : 25,446 '//visibility:public' : 21,962 'go_library' : 16,485 most common strings in Ruby: '\n' : 658 '' : 551 'shell' : 363 most common strings in Java: '' : 613 '\n' : 392 '0' : 293 most common strings in PHP: 'strict_param' : 1 'strict' : 1 'short_array_syntax' : 1
71. Thanks! @elithrar @francesc