Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Source, all alone, in the dark

Open Source, all alone, in the dark

How do you develop and advocate for an open source project you created by yourself? How does your project gain public recognition? How can you leverage your company's reputation?

These are personal observations of a maintainer.

Shlomi Noach

July 14, 2014
Tweet

More Decks by Shlomi Noach

Other Decks in Technology

Transcript

  1. About me • Engineer, DBA • Blog at http://openark.org •

    MySQL community member • Author of: ◦ openark-kit ◦ mycheckpoint (discontinued) ◦ common_schema ◦ Outbrain Propagator ◦ Outbrain Orchestrator • Occasional code contributor to MySQL - related projects. Bug reporter.
  2. openark-kit • Set of utilities for MySQL • A collection

    of python scripts which complement or enhance MySQL feature set • Modeled after Maatkit (predecessor of Percona Toolkit) • Notable tools: oak-online-alter-table, oak- chunk-update, oak-hook-general-log, oak- security-audit • Announced 2009 • To date: 2K-3K downloads per release
  3. Case study: oak-online-alter-table • Created and published early 2009 •

    Revolutionary, a real game changer • Allows for online, non-blocking, throttleable table refactoring. • For some 18 months, it seemed like I was its only production user
  4. Case study: oak-online-alter-table • About a year later, Facebook engineers

    called out for comments on their idea for online-schema-change. • After presenting oak-online-alter-table and some quick discussions, they adopted the general approach by openark-kit. • FB eventually forked and rewrote in PHP. • They say it revolutionized their deployment cycles. • Released as Facebook’s OSC
  5. common_schema • DBA’s framework for MySQL • A set of

    routines, views & scripting interpreter, completely residing within the database server • Distribution is a SQL file • Introduces QueryScript: SQL oriented scripting language ◦ QueryScript has the language constructs to support your common-yet -very-complex operations ◦ It is MySQL-aware, and mitigates locks and pitfalls commonly found in existing solutions
  6. common_schema • Introduced in 2011. • Released under GPLv2 •

    > 2.5K downloads (some 0.1k are mine :P) • Mature • Reviewed in High Performance MySQL 3rd Edition: "The common_schema is to MySQL as jQuery is to javaScript"
  7. Stuff that common_schema can do for you • Auto analyze

    and rotate your partitions • Find and eliminate redundant indexes • Audit your security setup • Find and destroy locking/locked transactions • Find and destroy blocking/idle transactions • Much much more...
  8. QueryScript code samples foreach($table, $schema, $engine: table in sakila) {

    if ($engine = 'InnoDB') ALTER TABLE :$schema.:$table ENGINE=InnoDB ROW_FORMAT=Compressed; }
  9. QueryScript code samples split ({start: 1200} : DELETE FROM sakila.rental

    WHERE rental_date < NOW() - INTERVAL 5 YEAR) { throttle 2; }
  10. Propagator & Orchestrator Releasing Outbrain’s internal tools • Designed as

    open source from day #1. • Fine balance: comply with company’s needs or generalize it for the greater audience? • Good to be public on the above. People like to know where the product is headed. They mostly like to know whether feature X will be supported. They like to know if X will not be supported.
  11. Propagator schema & data deployment tool that works on a

    multi-everything topology: • Multi-server: push your schema & data changes to multiple instances in parallel • Multi-role: different servers have different schemas • Multi-environment: recognizes the differences between development, QA, build & production servers • Multi-technology: supports MySQL, Hive (Cassandra on the TODO list) • Multi-user: allows users authenticated and audited access • Multi-planetary: TODO
  12. Orchestrator MySQL replication topology management and visualization tool • First

    of its kind • Visualize topologies, see what’s wrong • Refactor topologies; move slave around safely & visually • Supports command line, web API and web interface actions
  13. Releasing under Outbrain’s brand • Sure helps. • People like

    to know a large company uses a product on production. • Orchestrator got a lot of attention and positive feedback.
  14. The 3rd most time consuming task in OSS development (Proportional

    to the size of the task) Documentation! • People actually do read it. Make it as comprehensive as possible. • common_schema boasts some > 150 “man pages”. Documenting took by far more time resource than coding. • Proper documentation is the only thing that grants you the right to say: “RTFM!”
  15. The 2nd most time consuming task in OSS development (Proportional

    to the size of the task) Pick a License! • Consider Apache, GPL, BSD, MIT, WTFPL among others. • Use https://tldrlegal.com/ for human friendly license breakdown
  16. Links openark blog: http://openark.org Projects: ◦ https://acode.google.com/p/openarkkit/ ◦ https://code.google.com/p/common-schema/ ◦

    https://github.com/outbrain/propagator ◦ https://github.com/outbrain/orchestrator Twitter: @ShlomiNoach