open source Doing it by yourself All alone, in the dark Shlomi Noach

About me ● Engineer, DBA ● Blog at http://openark.org ● MySQL community member ● Author of: ○ openark-kit ○ mycheckpoint (discontinued) ○ common_schema ○ Outbrain Propagator ○ Outbrain Orchestrator ● Occasional code contributor to MySQL - related projects. Bug reporter.

openark-kit ● Set of utilities for MySQL ● A collection of python scripts which complement or enhance MySQL feature set ● Modeled after Maatkit (predecessor of Percona Toolkit) ● Notable tools: oak-online-alter-table, oak- chunk-update, oak-hook-general-log, oak- security-audit ● Announced 2009 ● To date: 2K-3K downloads per release

How you perceive your work

How others may perceive it

Case study: oak-online-alter-table ● Created and published early 2009 ● Revolutionary, a real game changer ● Allows for online, non-blocking, throttleable table refactoring. ● For some 18 months, it seemed like I was its only production user

I’m TELLING YOU, I’ve created a BEAST! You should be USING IT!

Case study: oak-online-alter-table ● About a year later, Facebook engineers called out for comments on their idea for online-schema-change. ● After presenting oak-online-alter-table and some quick discussions, they adopted the general approach by openark-kit. ● FB eventually forked and rewrote in PHP. ● They say it revolutionized their deployment cycles. ● Released as Facebook’s OSC

No content

No content

No content

No content

No content

Case study: oak-online-alter-table Your name matters. You big company name really matters.

common_schema ● DBA’s framework for MySQL ● A set of routines, views & scripting interpreter, completely residing within the database server ● Distribution is a SQL file ● Introduces QueryScript: SQL oriented scripting language ○ QueryScript has the language constructs to support your common-yet -very-complex operations ○ It is MySQL-aware, and mitigates locks and pitfalls commonly found in existing solutions

common_schema ● Introduced in 2011. ● Released under GPLv2 ● > 2.5K downloads (some 0.1k are mine :P) ● Mature ● Reviewed in High Performance MySQL 3rd Edition: "The common_schema is to MySQL as jQuery is to javaScript"

Stuff that common_schema can do for you ● Auto analyze and rotate your partitions ● Find and eliminate redundant indexes ● Audit your security setup ● Find and destroy locking/locked transactions ● Find and destroy blocking/idle transactions ● Much much more...

QueryScript code samples foreach($table, $schema, $engine: table in sakila) { if ($engine = 'InnoDB') ALTER TABLE :$schema.:$table ENGINE=InnoDB ROW_FORMAT=Compressed; }

QueryScript code samples split ({start: 1200} : DELETE FROM sakila.rental WHERE rental_date < NOW() - INTERVAL 5 YEAR) { throttle 2; }

Propagator & Orchestrator Releasing Outbrain’s internal tools ● Designed as open source from day #1. ● Fine balance: comply with company’s needs or generalize it for the greater audience? ● Good to be public on the above. People like to know where the product is headed. They mostly like to know whether feature X will be supported. They like to know if X will not be supported.

Propagator schema & data deployment tool that works on a multi-everything topology: ● Multi-server: push your schema & data changes to multiple instances in parallel ● Multi-role: different servers have different schemas ● Multi-environment: recognizes the differences between development, QA, build & production servers ● Multi-technology: supports MySQL, Hive (Cassandra on the TODO list) ● Multi-user: allows users authenticated and audited access ● Multi-planetary: TODO

Propagator ● Outbrain’s de-facto deployment tool Stack: MySQL, PHP, (yes, PHP, yes, I regret it), jQuery, Bootstrap

Orchestrator MySQL replication topology management and visualization tool ● First of its kind ● Visualize topologies, see what’s wrong ● Refactor topologies; move slave around safely & visually ● Supports command line, web API and web interface actions

Orchestrator Stack: MySQL, Go, Martini, D3, jQuery, Bootstrap

Releasing under Outbrain’s brand ● Sure helps. ● People like to know a large company uses a product on production. ● Orchestrator got a lot of attention and positive feedback.

The 3rd most time consuming task in OSS development (Proportional to the size of the task) Documentation! ● People actually do read it. Make it as comprehensive as possible. ● common_schema boasts some > 150 “man pages”. Documenting took by far more time resource than coding. ● Proper documentation is the only thing that grants you the right to say: “RTFM!”

The 2nd most time consuming task in OSS development (Proportional to the size of the task) Pick a License! ● Consider Apache, GPL, BSD, MIT, WTFPL among others. ● Use https://tldrlegal.com/ for human friendly license breakdown

Links openark blog: http://openark.org Projects: ○ https://acode.google.com/p/openarkkit/ ○ https://code.google.com/p/common-schema/ ○ https://github.com/outbrain/propagator ○ https://github.com/outbrain/orchestrator Twitter: @ShlomiNoach

Image credits https://www.flickr.com/photos/39679990@N05/8354263159 https://www.flickr.com/photos/mplonsky/853790788 https://www.flickr.com/photos/tambako/8494616379 https://www.flickr.com/photos/greencolander/3203922211 https://www.flickr.com/photos/bensonkua/2754312951 https://www.flickr.com/photos/battle_android/8264463438 https://www.flickr.com/photos/itspaulkelly/6662073403 https://www.flickr.com/photos/stellastella/5694418623/ https://www.flickr.com/photos/tanukifu/4454233952

Thank you! Questions?