Open Source, all alone, in the dark

How do you develop and advocate for an open source project you created by yourself? How does your project gain public recognition? How can you leverage your company's reputation?

These are personal observations of a maintainer.

1. open source Doing it by yourself All alone, in the dark Shlomi Noach
2. About me • Engineer, DBA • Blog at http://openark.org • MySQL community member • Author of: ◦ openark-kit ◦ mycheckpoint (discontinued) ◦ common_schema ◦ Outbrain Propagator ◦ Outbrain Orchestrator • Occasional code contributor to MySQL - related projects. Bug reporter.
3. openark-kit • Set of utilities for MySQL • A collection of python scripts which complement or enhance MySQL feature set • Modeled after Maatkit (predecessor of Percona Toolkit) • Notable tools: oak-online-alter-table, oak- chunk-update, oak-hook-general-log, oak- security-audit • Announced 2009 • To date: 2K-3K downloads per release
4. How you perceive your work
5. How others may perceive it
6. Case study: oak-online-alter-table • Created and published early 2009 • Revolutionary, a real game changer • Allows for online, non-blocking, throttleable table refactoring. • For some 18 months, it seemed like I was its only production user
7. I’m TELLING YOU, I’ve created a BEAST! You should be USING IT!
8. Case study: oak-online-alter-table • About a year later, Facebook engineers called out for comments on their idea for online-schema-change. • After presenting oak-online-alter-table and some quick discussions, they adopted the general approach by openark-kit. • FB eventually forked and rewrote in PHP. • They say it revolutionized their deployment cycles. • Released as Facebook’s OSC
14. Case study: oak-online-alter-table Your name matters. You big company name really matters.
15. common_schema • DBA’s framework for MySQL • A set of routines, views & scripting interpreter, completely residing within the database server • Distribution is a SQL file • Introduces QueryScript: SQL oriented scripting language ◦ QueryScript has the language constructs to support your common-yet -very-complex operations ◦ It is MySQL-aware, and mitigates locks and pitfalls commonly found in existing solutions
16. common_schema • Introduced in 2011. • Released under GPLv2 • > 2.5K downloads (some 0.1k are mine :P) • Mature • Reviewed in High Performance MySQL 3rd Edition: "The common_schema is to MySQL as jQuery is to javaScript"
17. Stuff that common_schema can do for you • Auto analyze and rotate your partitions • Find and eliminate redundant indexes • Audit your security setup • Find and destroy locking/locked transactions • Find and destroy blocking/idle transactions • Much much more...
18. QueryScript code samples foreach($table, $schema, $engine: table in sakila) { if ($engine = 'InnoDB') ALTER TABLE :$schema.:$table ENGINE=InnoDB ROW_FORMAT=Compressed; }
19. QueryScript code samples split ({start: 1200} : DELETE FROM sakila.rental WHERE rental_date < NOW() - INTERVAL 5 YEAR) { throttle 2; }
20. Propagator & Orchestrator Releasing Outbrain’s internal tools • Designed as open source from day #1. • Fine balance: comply with company’s needs or generalize it for the greater audience? • Good to be public on the above. People like to know where the product is headed. They mostly like to know whether feature X will be supported. They like to know if X will not be supported.
21. Propagator schema & data deployment tool that works on a multi-everything topology: • Multi-server: push your schema & data changes to multiple instances in parallel • Multi-role: different servers have different schemas • Multi-environment: recognizes the differences between development, QA, build & production servers • Multi-technology: supports MySQL, Hive (Cassandra on the TODO list) • Multi-user: allows users authenticated and audited access • Multi-planetary: TODO
22. Propagator • Outbrain’s de-facto deployment tool Stack: MySQL, PHP, (yes, PHP, yes, I regret it), jQuery, Bootstrap
23. Orchestrator MySQL replication topology management and visualization tool • First of its kind • Visualize topologies, see what’s wrong • Refactor topologies; move slave around safely & visually • Supports command line, web API and web interface actions
24. Orchestrator Stack: MySQL, Go, Martini, D3, jQuery, Bootstrap
25. Releasing under Outbrain’s brand • Sure helps. • People like to know a large company uses a product on production. • Orchestrator got a lot of attention and positive feedback.
26. The 3rd most time consuming task in OSS development (Proportional to the size of the task) Documentation! • People actually do read it. Make it as comprehensive as possible. • common_schema boasts some > 150 “man pages”. Documenting took by far more time resource than coding. • Proper documentation is the only thing that grants you the right to say: “RTFM!”
27. The 2nd most time consuming task in OSS development (Proportional to the size of the task) Pick a License! • Consider Apache, GPL, BSD, MIT, WTFPL among others. • Use https://tldrlegal.com/ for human friendly license breakdown
28. Links openark blog: http://openark.org Projects: ◦ https://acode.google.com/p/openarkkit/ ◦ https://code.google.com/p/common-schema/ ◦ https://github.com/outbrain/propagator ◦ https://github.com/outbrain/orchestrator Twitter: @ShlomiNoach
29. Image credits https://www.flickr.com/photos/39679990@N05/8354263159 https://www.flickr.com/photos/mplonsky/853790788 https://www.flickr.com/photos/tambako/8494616379 https://www.flickr.com/photos/greencolander/3203922211 https://www.flickr.com/photos/bensonkua/2754312951 https://www.flickr.com/photos/battle_android/8264463438 https://www.flickr.com/photos/itspaulkelly/6662073403 https://www.flickr.com/photos/stellastella/5694418623/ https://www.flickr.com/photos/tanukifu/4454233952
30. Thank you! Questions?