Kiba ETL: feedback on OSS / open-core sustainability for a Ruby gem

It’s pretty easy to get burned out in OSS, as we all know! Since starting working on Kiba ETL (a ruby data processing framework) in 2015, I made a number of decisions to avoid this at all cost, and make sure my research on this would be funded sustainably, in a way or another.

In this talk, I give feedback on which conscious design decisions & trade-offs I made over time and I will explain where I stand at the moment.

Transcript

1. ᇌ KIBA ETL
   Feedback on OSS / open-core sustainability for a Ruby gem
   — Thibaut Barrère, LoGeek SARL (https://www.logeek.fr)
   

   View Slide

2. ETL
   Extract Transform Load
   = a form of "Data Pipeline"
   

   View Slide

3. job = Kiba.parse do
   source MyCSVSource, filename: "files/input.csv"
   transform MyTransform, some_config
   transform MyOtherTransform, some_config
   transform { |row|
   ...
   }
   destination MyDatabaseDestination, database: db, table: :products
   end
   Kiba.run(job)
   

   View Slide

4. Sustainabilityies
   

   View Slide

5. Don't Compare Your Insides
   to Other People's Outsides
   

   View Slide

6. View Slide

7. 1995
   Demomaking. No GPU. No FPU. 486SX25
   

   View Slide

8. Funding! Support!
   ❤ thank you Mum & Dad
   Also VC Dad: "3D is nice, but will this earn money at some point?"
   

   View Slide

9. 2006
   Building datawarehouses (with Ruby)
   

   View Slide

10. ❤ activewarehouse-etl
    Declarative DSL for ETL — in Ruby!
    Loads of features!
    

    View Slide

11. source :in, {
    :file => 'extracted/crm-changes.csv',
    :parser => :delimited,
    :skip_lines => 1
    }, [:first_name, :last_name]
    after_read :ensure_fields_presence, { :fields => fields }
    transform(:id_partenaire) { |n,v,r| some_computation }
    destination :out, {
    :file => output_file,
    :include_headers => true,
    :separator => ";"
    }
    

    View Slide

12. 2011
    Open-source
    Maintenance
    

    View Slide

13. Loads of Features = Big Support Cost
    Time / Lost Income
    Applicability
    

    View Slide

14. Unsustainable
    

    View Slide

15. A Strict
    No Burnout
    & No Broke
    Policy
    

    View Slide

16. 3 OSS DECISIONS
    

    View Slide

17. #1: Reduce Scope
    Lessen the support cost
    

    View Slide

18. #2: Strive for
    Increased Applicability
    

    View Slide

19. #3: Time-Boxing
    

    View Slide

20. 2015Kiba ETL
    v0.5 (LGPL)
    

    View Slide

21. View Slide

22. Open-Source All The Thingz! No Broke, No Burnout!
    

    View Slide

23. Consulting
    & Internal Use
    

    View Slide

24. View Slide

25. View Slide

26. View Slide

27. Smaller OSS support cost ✔
    Increased OSS applicability ✔
    Time-boxing of OSS activity ✔
    

    View Slide

28. Funding! Support!
    Sustainable ✔
    

    View Slide

29. How to Keep Innovating?
    

    View Slide

30. Have a life! (outside coding)
    ❤
    ! "
    

    View Slide

31. Make Quality Time
    

    View Slide

32. Intellectual
    Property
    "Non Exclusive Rights"
    

    View Slide

33. View Slide

34. View Slide

35. A form of balance / sustainability
    

    View Slide

36. Questions?
    info@logeek.fr
    

    View Slide

37. Image credits
    https://commons.wikimedia.org/wiki/File:OricAtmos01a.jpg
    https://fr.wikipedia.org/wiki/Balance_Roberval#/media/Fichier:BalanceRobervalwhite.jpg
    https://giphy.com/gifs/sad-panda-Q3Ou3eFKMTFn2
    https://commons.wikimedia.org/wiki/File:ViennaConventionroadsignB2a.svg
    devil vs angel -> cannot find official author to credit so far
    https://twitter.com/SRUofPA/status/1029711272672149506
    https://www.pexels.com/photo/balance-macro-ocean-pebbles-235990/
    

    View Slide