How to Sell Elixir

http://www.elixir.london/Elixir-LDN-2017/evadne-wu

* * *

Technical complexity is now a major cost of doing business and the Elixir/Erlang ecosystems offer appropriate tools to efficiently manage complexity. In its realisation we have built a product entirely based on this technology, and now have stories to share.

In this talk, Evadne will attempt to cover:

1. Their economic case for adopting this particular ecosystem, from both a development velocity point of view and a maintenance/cost point of view.
2. Problems they have faced and solutions they have come up with, both on an organisational level and a technical level, in realising the project.
3. Some ideas for technical leaders wishing to adopt Elixir for their next greenfield or brownfield project.

Talk objectives:
To provide the audience with an appropriate evaluation framework for candidate technologies (i.e. boss persuasion)
To provide a collection of anecdotes on our adoption of the Erlang/OTP and Elixir ecosystems for the benefit of the community (i.e. war stories)
To identify areas for further enhancement both within the technology stack, and around how the stack is enhanced. (i.e. work suggestions)
Target audience:
This talk is intended for technical team leaders and senior software engineers wishing to transition into a role with some more management responsibilities; it is also best suited for relatively small teams (5-25 developers) that have prior experience in developing web applications, since that is the audience who shall find the views most relatable.
Evadne is a London-based technologist with a passion for data visualisation and experience in technical management. She has experience in software architecture, application prototyping, and system operations management. Evadne presently serves as the Head of Exam Systems for Faria Education Group, which is an international education SaaS company with staff in UK, EU and USA.

Transcript

1. How to Sell Elixir
   Evadne Wu
   
   Head of Exam Systems, Faria Education Group
   ev@radi.ws / github.com/evadne
   last updated
   
   16 August 2017
   

   View Slide

2. Outline
   ➤ Takeaway (Precepts)
   ➤ Team & Organisation Background
   ➤ Our First & Second Contacts with OTP
   ➤ Lessons Learned from Transitioning to OTP
   ➤ Closing Thoughts
   

   View Slide

3. Precepts
   

   View Slide

4. Boehm [1983]
   

   View Slide

5. Risk / Value
   When choosing technologies, you do not always have the luxury of
   hindsight. Therefore it is always a matter of managing risk.
   Optimise for different goals in different phases:
   ➤ Raise Potential Value by proper inception/design
   ➤ Reduce Value at Risk by proper execution
   ➤ Reduce impact to Value by designing for operations / maintenance
   

   View Slide

6. Complexity
   SaaS Adage: More Features, More Customers
   ➤ Not exactly true
   However,
   ➤ Complexity ≠ Profitability
   ➤ Resource Requirements ∝ Complexity
   

   View Slide

7. Leverage
   Resource Requirements ∝ (Effort / Leverage).
   ➤ With higher Leverage, you can do more with less i.e. ship earlier.
   ➤ Shorter Time Horizon can reduce Risk.
   NB: Good technology can’t save you from bad specification
   ➤ But finding out about it earlier also reduces Risk
   

   View Slide

8. Background
   

   View Slide

9. The Organisation
   SaaS Provider for the International Education sector
   ➤ Established in 2007
   ➤ 39–67% market share in our niche segments
   ➤ 2,300 customers in 120 countries
   ➤ Goal: Support the Transition From Paper
   ➤ We offer Software with a Human Touch; no IVRs ever
   

   View Slide

10. The Team
    UK branch for Managed Services
    ➤ We build and operate applications for strategically important clients
    ➤ And we help the rest of the organisation with intersectional work
    ➤ Business Analysis and Specification-Writing
    ➤ Service Development
    ➤ Not Maintenance
    

    View Slide

11. First & Second Contacts
    

    View Slide

12. Shifting Business Requirements
    We can capture more of the market by becoming an integrator
    The downside of being an integrator is having to integrate
    So, one day we wrote a message broker in Ruby…
    ➤ Classic Message Bus architecture
    ➤ Messages, Queues hosted by RabbitMQ
    ➤ Processing, Fanout and Reconciliation in CRuby
    

    View Slide

13. Some of The Many Problems
    Spawning one OS process for each Customer is highly inefficient
    ➤ Threading in CRuby is still a mystery
    Process monitoring seems easy, but not when you need to be dynamic
    Work distribution among nodes is also not very easy
    ➤ SELECT * FROM schools WHERE host = ‘x’
    The system dies completely on un-caught exceptions
    ➤ Error handling logic everywhere
    

    View Slide

14. Solutions?
    We went and made two more prototypes
    ➤ JRuby + custom supervision logic + true threads via Concurrent Ruby
    ➤ Elixir + OTP magic + dynamic supervision tree
    Ultimately we did not have enough time to do what we wanted
    ➤ We signed a contract with another vendor to deal with this for us
    

    View Slide

15. Postmortem
    We would not have been able to make the original solution work
    ➤ Its performance characteristics are not in the bracket we require
    However, we learned a lot from the two other prototypes
    ➤ The JVM ecosystem is a very good alternative when you need things done
    ➤ The OTP ecosystem has better tooling for a few specific things
    ➤ Especially for orchestrating highly concurrent, interactive workloads
    

    View Slide

16. Second Contact
    Vendor B, which we use, gets bought by Vendor C
    ➤ Hello, here is the new version with a totally different API…
    ➤ …you must store all the files with us now…
    ➤ …yes, there is a migration tool but it does not move metadata…
    ➤ …the cost estimate is $x,000 per month, we hope you’d like it…
    

    View Slide

17. Second Contact
    Our VPE is not having any of it
    ➤ Start sending out tenders and getting replies from other Vendors
    ➤ Big “Buy vs Build” decision starting to brew in VPE’s head
    

    View Slide

18. Vendors Start to Reply…
    ➤ “We do not have this feature — it will cost $100,000 or so extra, and we
    will try to build this in for you specifically, if you sign the dotted line.”
    ➤ “We do not offer trials but we can send you a copy once you sign.”
    ➤ “Download a test version here!” (The Download site is broken.)
    

    View Slide

19. Adventures in “Buy vs Build” World
    It was difficult to get even a stable (predictable) quote or even a
    testable artefact without committing sizeable chunks of money.
    ➤ Complete antithesis of how we operate
    ➤ We maintain a full-featured demonstration environment for anybody who
    visits the link — no information required.
    ➤ Unnecessarily drains our Research & Professional Development budget
    

    View Slide

20. Looking at Options
    We are at the crossroads looking at one of three options
    ➤ New Custom-Built component
    ➤ New COTS component
    ➤ Another SaaS component
    

    View Slide

21. Mutual Realisation (×2)
    Custom integration work is completely inevitable.
    A team that is in-house is much easier to work with, than one that
    needs to serve many customers with potentially conflicting
    requirements.
    ➤ As an ISV ourselves, we have our own Decision Framework shared with
    customers explaining how we approach new feature requests.
    ➤ Roughly based on popularity, priority, and future applicability.
    

    View Slide

22. Our Proposal
    Take what we have learned from dealing with >1m documents.
    Make a proper service which serves our own use cases.
    Demonstrated lower TCO — vs. COTS/SaaS
    ➤ Also considering Communication/Coordination Cost
    Demonstrated Feasibility with end-to-end prototype
    

    View Slide

23. “Just Get It Done”
    To ensure prompt decision, our Proposal comes with a Prototype
    ➤ Fully working for the high impact use cases
    ➤ Fully deployed via AWS CFN
    ➤ We were then told to “get it done”. Sold!
    

    View Slide

24. Transitioning to OTP
    

    View Slide

25. Components
    From To Results
    Rails Phoenix
    Assets Pipeline Brunch ⏳→→
    ActionCable/Faye Phoenix Channels
    ActiveRecord Ecto
    

    View Slide

26. Tasks
    Task From To Results
    AWS Interop Fog ExAWS
    Authentication Devise Guardian
    Thumbnail
    Generation
    CarrierWave
    Custom C
    Program
    ⏳→→
    

    View Slide

27. Closing Thoughts
    

    View Slide

28. Precepts
    Most of the problems can be looked through the lenses of…
    ➤ Risk Assessment and Management: Long-Term vs One-Shot
    ➤ Nature of the Problem: Wicked vs Righteous
    ➤ Team Organisation: Mechanical vs Organic
    ➤ Modes of Control: Free Market, Contractural vs Cultural
    

    View Slide

29. Grove [1983]
    

    View Slide

30. Objections / Antidotes
    Perception Reality Antidote Prerequisite
    “New (Unproven)”
    Implementation
    Risk
    Early End-to-End
    Prototype
    Freedom to Explore
    & Develop
    “Non-Standard”
    IT Governance
    Complexity
    Bottom-Up

    Cultural Change
    Explicit KPIs and
    Past Successes
    “Obscure”
    Recruiting &

    Maintenance Risk
    Professional
    Development
    Community
    Engagement
    

    View Slide