We open sourced our trade secrets, and so can you!

AppNeta TraceView is distributed tracing software as a service. Our Ruby instrumentation is a gem that hooks into popular frameworks, clients, and drivers to capture performance data from a running application – and if you think storing data in the cloud is a hard sell, just wait until you mention that you're modifying executing code to capture it! Bad instrumentation could have all kinds of adverse impacts on your application, from crashes to slowdowns to subtle corruptions, so our guiding principle is "first, do no harm". Plenty of customers have found our gem to be fast and stable under real-world conditions, but until recently, they had to take us at our word that we weren't shipping backdoors, ticking time bombs, or dark magic.

About a year ago, we decided that it was time to open source our gem. Starting off proprietary meant that we'd never planned for the world to read our code, so we didn't yet understand what risks we were exposing ourselves and our customers to. It was also internal-focused, so we weren't even sure whether anyone who wasn't an AppNeta engineer would get anything out of it. But we wanted to make it happen, and for it to happen to more code in the future, so we developed an internal process for open-sourcing code sustainably. Before launch day, we got buy-in from involved departments, vetted the license with our lawyers, figured out how our support team would support it, audited code and removed information that needed to stay private, wrote tests that anyone could run, turned internal documentation into "TFM", figured out our internal workflow, and developed a framework for community contributions. It took a few months to get everything in order, but the day finally came when we switched over our RubyGems build to the new repo and opened it to the world. (I wrote the announcement blog post – yay!)

Our gem has been on GitHub at appneta/oboe_ruby for over six months now, and we're very pleased with the results we've seen. We've had no problem continuing to sell our product, our Ruby customers have been much more engaged, and we've even been able to release new instrumentation based on their contributed code. We're glad we open sourced our trade secrets – what's stopping you?

Transcript

1. We  open  sourced  our  trade   secrets,  and  so  can  you!   James  Meickle   Developer  evangelist,  AppNeta   @jmeickle     Boston  Ruby  on  Rails  Meetup   August  26,  2014  
2. AppNeta   •  ApplicaFon   performance   management  (APM)   –  Client   –  Network   –  Server   •  SoNware  as  a  service   •  Cloud  hosted  
3. A  brief  history  of  APM   •  Closed  source   monitoring  for  closed   source  applicaFons   •  Oriented  around   mainframes  and  J2EE   •  Incredibly  high  cost,  not   producFon-­‐ready   •  Today:  rapid  value,  easy   deployment,  low  cost  
4. AppNeta  TraceView   •  Distributed  tracing   (Zipkin,  Dapper)   •  Bytecode   instrumentaFon   •  Java,  .NET,  PHP,  Ruby,   Python,  node.js   •  Definitely  not   Tracely/cs!  
5. THE  GEM   ImplemenFng  TraceView’s  funcFonality  in  Ruby  
6. Our  maintainer   •  Peter  Lombardo  came   on  a  few  years  ago   •  Exclusively  focused  on   the  oboe  gem   •  Works  remote  from   Italy,  or  else  he’d  be   giving  this  talk!  
7. oboe   •  Installed  like  a  normal   gem  (with  C  extension)   •  No  configuraFon  required   •  Provides:   –  ApplicaFon  performance   management   –  SoNware  as  a  service   –  Distributed  tracing   –  Bytecode  instrumentaFon   –  Trace  API  
8. ApplicaFon  performance  management   •  Collect  performance  data:   –  Database  queries   –  Cache  requests   –  Remote  calls   –  TemplaFng  Fme   •  Without  causing   problems:   –  IdenFcal  behavior   –  Overhead  below  1%   –  Stable  
9. SoNware  as  a  service   •  Don’t  do  any  processing   locally   •  Report  data  to  the   agent  (UDP)   •  Report  data  to  collector   (SSL,  on  Heroku)  
10. Distributed  tracing   •  Follow  calls  from  one   applicaFon  to  another   –  Different  servers   –  MulFple  languages   –  Asynchronous   •  Common  request/ response  protocols:   –  HTTP   –  ThriN   –  EJB  (for  JRuby)  
11. Bytecode  instrumentaFon   •  Support  common   components:   –  Frameworks   –  Libraries   –  Drivers/clients   •  Without  code   modificaFon   –  Ruby,  Python:  monkey   patching   –  Java:  classloader   –  PHP:  L  
12. Trace  API   •  Same  API  that  we  use   when  adding  layers   •  Duck  typing  for  events     *please  don’t  report  credit  card  numbers,  thanks   •  Our  backend  is  built   around  supporFng  it   •  Report  anything!*  
13. Example  1:  memcache  
14. Example  2:  rack  
15. Example  2:  Net::HTTP  
16. Example  4:  resque  
17. THE  PROCESS   Opening  the  oboe  gem  to  the  world  
18. Why  did  we  open  source  oboe?   •  We  <3  open  source   •  Larger  customers  want   to  audit  our  code   •  Increased  community   contribuFons   •  Beher  capture  of   reported  issues   •  Good  publicity  J  
19. Why  couldn’t  we  “flip  the  switch”?   •  The  code  was  not   appropriately   documented   •  The  test  suite  needed   work   •  There  was  no   contribuFon  guide   •  It  wasn’t  licensed   •  Not  branded  properly!   •  Issues  contained   customer-­‐specific   informaFon  
20. DocumentaFon   •  Updated  comments  for   specific  funcFons   •  Added  overarching   documentaFon  too   •  Had  a  few  people  take  a   pass  at  improving  it   •  Updated  the  support   portal,  too  J  
21. Tests   •  “Before  opensourcing,  we  had  to   move  away  from  manual  tesFng   via  nosetests  (30  something  stack   variaFons,  manual  boot  and   manually  run  nosetests,  rinse  and   repeat  for  the  next  hour).     We  went  with  Minitest.  Today   with  Minitest,  we  now  test  7   Ruby  versions:  each  with  202   tests,  1979  asserFons  -­‐  biggest   win  beyond  coverage  is  that  it’s   automated  with  Travis  for  each   git  commit.”   –  Peter  Lombardo  
22. ContribuFon  guide   •  External   documentaFon:  let   people  know  how  to   write  good  code   •  Internal  policies:  how  to   merge  (or  reject!)  code   from  outside  authors   •  Had  to  involve  the   support  team!  J  
23. Licensing   •  We  have  open  source   instrumentaFon,  but  it’s   not  free  soNware   •  Some  parts  of  our  stack   will  always  be  proprietary   •  CompeFFve  advantage  is   a  huge  concern  in  this   industry   •  But  this  is  sFll  an   unfathomable  change   from  where  we  were  10   years  ago  J  
24. Branding   •  We  were  already   AppNeta  TraceView   •  But  we  sFll  were   TracelyFcs  in  our  code   •  Not  as  simple  as     s/foo/bar/;  lots  of   unexpected  places   (support  URLs!)  
25. Issues  with  issues   •  We  didn’t  want  to  lose   historical  issues   •  New  issues  would  also   need  to  reference   specific  customers  and   post  about  their  data  
26. The  soluFon   •  “The  key  was  to  use  two  repos  (1   private  and  1  public)  using  two  git   remotes.    Now  our  default  is  that   all  issues  are  filed  in  the  public   repository  unless  it  contains   customer  specific  data.    If  a  public   issue,  turns  out  to  need  a  pointer   to  some  internal  data  or   resource,  we  file  a  corresponding   internal  issue  that  points  to  the   public  issue.    We  now  always   prefer  the  public  repo  over  the   private  and  only  use  the  private   when  absolutely  necessary.”     –  Peter  Lombardo  
27. GitHub  issues   •  Work  being  done  lives   in  GitHub  issues  on  the   public  repo   •  Anyone  can  post  there!   •  For  long  term  features,   we  may  close  issues  and   move  them  into  Asana  
28. The  final  countdown  
29. MarkeFng  helped!   •  Pre-­‐launch   –  ImplicaFons  of  licensing   –  How  to  reward  and   publicize  contributors   –  Info  page  for  repo   •  Post-­‐launch   –  Launch  blog  post   –  Social  media   –  ExisFng  customers!  
30. Launch  day   •  Set  up  the  public  repo   as  a  private  one   •  Switch  our  build   process  (rubygems)  to   use  the  right  repo   •  Post  the  new  build  as  a   GitHub  Release  too   •  Publish  the  blog  post!  
31. THE  RESULT   What  we’ve  seen  aNer  open  sourcing  
32. appneta  /  oboe-­‐ruby   •  Not  many  visitors  L   •  But  they  were  incredibly   engaged:  more  than  25%   of  them  cloned  the  repo.   •  And  about  a  third  of  them   contributed  code!   •  If  the  rest  of  our  site   converted  that  well,  we’d   sell  $1,000,000  more   every  year  J  
33. Beher  support  for  powerusers   •  The  trace  API  is   incredibly  powerful   •  But  it’s  also  hard  to  use   correctly;  most  people   don’t  manage  to  do  so   •  We  only  noFce  when   they  file  support  Fckets   •  Post  custom  code  on   GH,  and  we  can  help   while  you  write  it!  J  
34. More  contributors   •  80%:  post  an  issue   •  15%:  post  a  fix/snippet   •  5%:  an  enFre  feature!   –  Grape  API:  Todd  Lunter   at  Swipely   –  EventMachine:  Diogo   Benica  at  Abril  Midia   •  Anyone  who  codes  a   feature  gets  one  year  of   TraceView  Enterprise  J  
35. Happier  maintainer   •  “Not  sure  if  this  part  is  going  to  make   sense  but  working  on  an  opensource   project  is  like  having  a  weight  liNed   off  of  your  shoulders.    When  it’s   private,  proprietary  code,  the  burden   is  completely  on  you  to  assure  quality   and  to  not  screw  up.    When  the  code   is  opensource,  there  is  a  slightly   calming  feel  because  you  know  that   your  code  is  public  and  is  available  to   be  reviewed  and  improved  upon  by   anyone.    We  love  the  contribuFons   we’ve  gohen  so  far  and  are  always   looking  for  more.”   –  Peter  Lombardo  
36. Progress  on  the  gem   •  Being  open  source  has   not  impacted  our  ability   to  ship  code  at  all  –  yay!   •  In  fact,  we’ve  greatly   enhanced  our  Ruby   coverage   •  Mandatory  plug:  check   out  our   enhanced  JRuby   support  J  
37. Thank  you!   Come work for us! We’re a mostly Python shop J" " You can meet us at:" •  Sep 15-17: Velocity New York" •  Sep 18: WebPerfDays New York" •  Oct 21: TechBreakfast New York" •  October 24: Surge (DC)" •  Nov 11-14: AWS re:Invent (Vegas!)" Or just try us out:" hhp://www.appneta.com/products/traceview/