The Open Source
Developer’s Toolbox
Open Source 101, February 2017
Tyler Langlois
Infrastructure Engineer, Elastic
Slide 2
Slide 2 text
$ whois tylerjl
● Infrastructure engineering/devops
at Elastic
○ Operations and software dev
● Other pursuits
○ Overwatch Zenyatta main,
has a Mork-chi named Morty
normal
person
Slide 3
Slide 3 text
Who is This For?
● What OSS tools are out
there?
● What are they good
for?
● Should I pick x or y?
● What are people
using?
Background
Elasticsearch,
logstash
Kibana
Beats
Logstash
Infra/Devops
● CI
● Dependencies
● Toolchains
● Distribution
Slide 7
Slide 7 text
Languages
Slide 8
Slide 8 text
Languages
Stricter, more
controlled
Dynamic, diverse
libraries, variety
of use cases
Specialized,
well-suited for
particular uses
Slide 9
Slide 9 text
Writing
Slide 10
Slide 10 text
Code and documentation
● Remember, optimize for your productivity
● Bear in mind some languages (Java?) will dictate an effective
environment (i.e., IntelliJ debugger)
● OS isn’t a hard prerequisite for many authoring options
● Even OS-specific dev technologies (see Docker) are going
cross-platform (ish)
Writing
Slide 11
Slide 11 text
Writing
CLI
● +
○ Lightweight
○ Composable
● -
○ Features like refactoring
○ Learning curve
IDE
● +
○ Featureful
○ Tight language integration
● -
○ Complexity
○ Obfuscation
To IDE or not IDE?
Slide 12
Slide 12 text
Writing
CLI IDE
To IDE or not IDE?
Eclipse
Atom
Language-specific (pycharm,
rubymine, etc.)
Vim
Emacs
nano… I guess?
Slide 13
Slide 13 text
Writing
Vim
● Verb-like editing
● Standard
● Revitalized with Neovim
● Diverse range of language
plugins
● Arcane for beginners
● Feat. like refactoring = poor
Slide 14
Slide 14 text
Writing
Emacs
● Strong, pluggable codebase
● Hotkey-driven
● Equally diverse plugin
community as Vim
● Added functionality (IRC, etc.)
● Arcane for beginners
● Feat. like refactoring = poor
Slide 15
Slide 15 text
Writing
Eclipse
● Tight language integration
● Pluggable
● Integrates SCM
● Strong support for langs like
Java
● Can become resource hog
● Hides details under the covers
Slide 16
Slide 16 text
Writing
Atom
● Latest addition, .js based
● Somewhere between CLI and
IDE load
● Sponsored + Developed at
Github
● +/- Javascript
Slide 17
Slide 17 text
Writing
Traditional Tools
● SVN, CVS
○ Remote SCM server
○ Distributed development
happens via locking
○ Only relevant pieces
checked out
Modern Replacements
● Git, Mercurial
○ Server + local components
separate
○ Lightweight branching
○ Distributed development
○ De facto OSS winner
Source Control
Slide 18
Slide 18 text
Writing
Source Control
Slide 19
Slide 19 text
Writing
Git, Mercurial
● Immutable and graph-like history
● Interface options
○ IDE, CLI, dedicated software
● Git vs. Mercurial?
○ git: low-level swiss army
knife
○ hg: more user-friendly, less
adoption
Slide 20
Slide 20 text
Writing
Git, Mercurial
● Native tools aren’t bad
● CLI / IDEs create identical histories;
users can collaborate with either
● It’s worthwhile to brush up on the
data model to avoid major blowups
(Google “the git parable”)
Slide 21
Slide 21 text
Utilities
Slide 22
Slide 22 text
Language + Editor + SCM + … ?
● Additional command-line tools?
○ Homebrew, chocolatey, $package_manager
● Shell proficiency
○ Bash is standard, but beefier options are available
○ Zsh, fish
Utilities
Slide 23
Slide 23 text
Language + Editor + SCM + … ?
● Language version manager
○ Versions differ significantly, many projects require a specific
major version
Utilities
jenv, sdkman
nvm
Rvm,
rbenv
virtualenv
stack
… ?
Slide 24
Slide 24 text
Services
Slide 25
Slide 25 text
Services
Why?
Where do you:
● Test the software?
● Build the software?
● Run the software?
● Document the software?
● Collaborate on the software?
The sad reality: there is no free lunch
Slide 26
Slide 26 text
Services
A Word of Warning:
Slide 27
Slide 27 text
Services
Collaboration - Cloud
GitHub
● De facto OSS repository
● Git (no mercurial)
● Model is pricing for private
repos
Bitbucket
● Smaller but also popular
● git + mercurial
Slide 28
Slide 28 text
Services
Collaboration - Self-Hosted
GitLab
● Self-hosted GitHub
alternative
gogs
● Self-contained go service
gitolite
● CLI-based service
Slide 29
Slide 29 text
Services
Testing
$ rspec # ruby
$ py.test # python
$ go test # go
...many more
Slide 30
Slide 30 text
Services
Testing
Travis CI (Cloud)
● Free for OSS
● Tightly integrated into
GitHub
Jenkins (Self-hosted)
● Most common self-hosted
option
Slide 31
Slide 31 text
Services
Testing
Slide 32
Slide 32 text
?
Slide 33
Slide 33 text
Databases
Services
Slide 34
Slide 34 text
Databases
Services
1. Don’t try and solve Google-sized scaling issues first
2. SQL isn’t uncool
3. It’s all about the shape of your data
4. Know your access patterns
5. SAFETY
6. ???
7. Profit
Slide 35
Slide 35 text
Databases
Services
● SQL
○ Postgresql, MySQL
○ Not hip, but been around a long time
○ Why are they still relevant?
■ Lots of data is relational!
○ SQLite convenient option for local development
○ Well-documented for self-hosting or hosted (RDS)
solutions
Slide 36
Slide 36 text
Databases
Services
● Search
○ Lucene is the grandaddy
○ Solr, Elasticsearch
○ RESTful, JSON-oriented, distributed
○ Useful to make an application:
■ Autocomplete
■ Highlight
■ Real fast
Slide 37
Slide 37 text
Databases
Services
● Timeseries
○ For when you’ve just got numbers
○ Tuned to numerics
Slide 38
Slide 38 text
Databases
Services
● Other NoSQL
○ MongoDB
■ Easy to get started, scalable
○ Hadoop Ecosystem
■ For really, REALLY big data
■ HDFS, MapReduce, etc.
○ + many more