* * *
Technical complexity is now a major cost of doing business and the Elixir/Erlang ecosystems offer appropriate tools to efficiently manage complexity. In its realisation we have built a product entirely based on this technology, and now have stories to share.
In this talk, Evadne will attempt to cover:
1. Their economic case for adopting this particular ecosystem, from both a development velocity point of view and a maintenance/cost point of view.
2. Problems they have faced and solutions they have come up with, both on an organisational level and a technical level, in realising the project.
3. Some ideas for technical leaders wishing to adopt Elixir for their next greenfield or brownfield project.
To provide the audience with an appropriate evaluation framework for candidate technologies (i.e. boss persuasion)
To provide a collection of anecdotes on our adoption of the Erlang/OTP and Elixir ecosystems for the benefit of the community (i.e. war stories)
To identify areas for further enhancement both within the technology stack, and around how the stack is enhanced. (i.e. work suggestions)
This talk is intended for technical team leaders and senior software engineers wishing to transition into a role with some more management responsibilities; it is also best suited for relatively small teams (5-25 developers) that have prior experience in developing web applications, since that is the audience who shall find the views most relatable.
Evadne is a London-based technologist with a passion for data visualisation and experience in technical management. She has experience in software architecture, application prototyping, and system operations management. Evadne presently serves as the Head of Exam Systems for Faria Education Group, which is an international education SaaS company with staff in UK, EU and USA.
How to Sell Elixir
Head of Exam Systems, Faria Education Group
[email protected] / github.com/evadne
16 August 2017
➤ Takeaway (Precepts)
➤ Team & Organisation Background
➤ Our First & Second Contacts with OTP
➤ Lessons Learned from Transitioning to OTP
➤ Closing Thoughts
Risk / Value
When choosing technologies, you do not always have the luxury of
hindsight. Therefore it is always a matter of managing risk.
Optimise for different goals in different phases:
➤ Raise Potential Value by proper inception/design
➤ Reduce Value at Risk by proper execution
➤ Reduce impact to Value by designing for operations / maintenance
SaaS Adage: More Features, More Customers
➤ Not exactly true
➤ Complexity ≠ Proﬁtability
➤ Resource Requirements ∝ Complexity
Resource Requirements ∝ (Effort / Leverage).
➤ With higher Leverage, you can do more with less i.e. ship earlier.
➤ Shorter Time Horizon can reduce Risk.
NB: Good technology can’t save you from bad speciﬁcation
➤ But ﬁnding out about it earlier also reduces Risk
SaaS Provider for the International Education sector
➤ Established in 2007
➤ 39–67% market share in our niche segments
➤ 2,300 customers in 120 countries
➤ Goal: Support the Transition From Paper
➤ We offer Software with a Human Touch; no IVRs ever
UK branch for Managed Services
➤ We build and operate applications for strategically important clients
➤ And we help the rest of the organisation with intersectional work
➤ Business Analysis and Speciﬁcation-Writing
➤ Service Development
➤ Not Maintenance
First & Second Contacts
Shifting Business Requirements
We can capture more of the market by becoming an integrator
The downside of being an integrator is having to integrate
So, one day we wrote a message broker in Ruby…
➤ Classic Message Bus architecture
➤ Messages, Queues hosted by RabbitMQ
➤ Processing, Fanout and Reconciliation in CRuby
Some of The Many Problems
Spawning one OS process for each Customer is highly inefﬁcient
➤ Threading in CRuby is still a mystery
Process monitoring seems easy, but not when you need to be dynamic
Work distribution among nodes is also not very easy
➤ SELECT * FROM schools WHERE host = ‘x’
The system dies completely on un-caught exceptions
➤ Error handling logic everywhere
We went and made two more prototypes
➤ JRuby + custom supervision logic + true threads via Concurrent Ruby
➤ Elixir + OTP magic + dynamic supervision tree
Ultimately we did not have enough time to do what we wanted
➤ We signed a contract with another vendor to deal with this for us
We would not have been able to make the original solution work
➤ Its performance characteristics are not in the bracket we require
However, we learned a lot from the two other prototypes
➤ The JVM ecosystem is a very good alternative when you need things done
➤ The OTP ecosystem has better tooling for a few speciﬁc things
➤ Especially for orchestrating highly concurrent, interactive workloads
Vendor B, which we use, gets bought by Vendor C
➤ Hello, here is the new version with a totally different API…
➤ …you must store all the ﬁles with us now…
➤ …yes, there is a migration tool but it does not move metadata…
➤ …the cost estimate is $x,000 per month, we hope you’d like it…
Our VPE is not having any of it
➤ Start sending out tenders and getting replies from other Vendors
➤ Big “Buy vs Build” decision starting to brew in VPE’s head
Vendors Start to Reply…
➤ “We do not have this feature — it will cost $100,000 or so extra, and we
will try to build this in for you speciﬁcally, if you sign the dotted line.”
➤ “We do not offer trials but we can send you a copy once you sign.”
➤ “Download a test version here!” (The Download site is broken.)
Adventures in “Buy vs Build” World
It was difﬁcult to get even a stable (predictable) quote or even a
testable artefact without committing sizeable chunks of money.
➤ Complete antithesis of how we operate
➤ We maintain a full-featured demonstration environment for anybody who
visits the link — no information required.
➤ Unnecessarily drains our Research & Professional Development budget
Looking at Options
We are at the crossroads looking at one of three options
➤ New Custom-Built component
➤ New COTS component
➤ Another SaaS component
Mutual Realisation (×2)
Custom integration work is completely inevitable.
A team that is in-house is much easier to work with, than one that
needs to serve many customers with potentially conﬂicting
➤ As an ISV ourselves, we have our own Decision Framework shared with
customers explaining how we approach new feature requests.
➤ Roughly based on popularity, priority, and future applicability.
Take what we have learned from dealing with >1m documents.
Make a proper service which serves our own use cases.
Demonstrated lower TCO — vs. COTS/SaaS
➤ Also considering Communication/Coordination Cost
Demonstrated Feasibility with end-to-end prototype
“Just Get It Done”
To ensure prompt decision, our Proposal comes with a Prototype
➤ Fully working for the high impact use cases
➤ Fully deployed via AWS CFN
➤ We were then told to “get it done”. Sold!
Transitioning to OTP
From To Results
Assets Pipeline Brunch ⏳→→
ActionCable/Faye Phoenix Channels
Task From To Results
AWS Interop Fog ExAWS
Authentication Devise Guardian
Most of the problems can be looked through the lenses of…
➤ Risk Assessment and Management: Long-Term vs One-Shot
➤ Nature of the Problem: Wicked vs Righteous
➤ Team Organisation: Mechanical vs Organic
➤ Modes of Control: Free Market, Contractural vs Cultural
Objections / Antidotes
Perception Reality Antidote Prerequisite
Freedom to Explore
Explicit KPIs and