Phoenix after 100,000 lines Renan Ranelli

whoami Renan Ranelli (also known as Milhouse) Senior Software Engineer @ (previously worked with Ruby & C# mainly)

whoami Renan Ranelli (also known as Milhouse) Senior Software Engineer @

Objectives ➔ Share my last 2 years experiences Working with Elixir in a "big" app ➔ Present & address concerns People have when considering Elixir for more "critical" projects ➔ Convince you it's great (& safe!) to use Elixir & Phoenix for big projects

First, a little context: What are we talking about?

Our context - Early stage startup (money was short) - B2B SaaS, "Enterprise application". We're not a "scalability" case. Sry. - Early stage ecosystem -- june 2015 - Elixir: 1.0.5 (1.5.1 today) - Phoenix: 0.14 (1.3.0 today) - Ecto: 0.13.1 (2.2.1 today) - No previous experience with Elixir in the "founding team"

Why we chose Elixir?!

Why we chose Elixir?! - Because it looks like Ruby? - No - Because it is super fast? - No - Because we needed to scale to a billion users overnight? - No… if we scaled to a million we would be rich - Because we were spending a lot of money on machines? - No. We didn't even have a product! - Because we understood the Erlang platform, weighted its pros and cons and found out that it was the best fit for our needs? - No. Definitely No.

Why we chose Elixir?! **Hireability**

Hireability We wanted a small but very effective & invested team. We then decided to use Elixir as a lever: "We are going to give you a great opportunity to work full time with Elixir & incredible people"

Some (common) concerns What are the risks of that choice?

Our fears concerns - Will we find libraries & other ecosystem related stuff? Will we need to build the world? - Will this whole thing turn into abandon-ware? - How productive is this framework? Should we be settling for something like Rails instead? - Will this be good for "complex" business logic? (which we do have lots) - Will this be stable in production? - (more particular to our case:) Will our hiring strategy work?

Our conclusion then: We won't know unless we try. Maybe in 2 years we'll be able to answer that

As expected, we had problems along the way.

Interacting with other databases - Interacting with "non-conventional" databases: In the beginning, we couldn't find good libraries to interact with Cassandra and ElasticSearch. - We ended up writing our own... - Today, we've got Xandra for Cassandra, but I don't know of any good alternative for ElasticSearch. Ping me if you do.

Authentication, Accounts, Users ... - At the time, we couldn't find anything ready for use… - We ended up writing our own... - Today, we've got a whole section in awesome elixir...

Massive Recompilations - With more than a thousand modules, we discovered that it is quite easy to get in a situation where a big chunk of the project is recompiled after changing just one file. - We suffered a lot, but got to understand the causes... - See http://milhouseonsoftware.com/2016/08/11/understanding-elixir-recompilation/ or my (unreleased) empex.co talk on this theme ~ demo ~

Timex many major updates - We started with Timex 0.15, and we're on 3.1.24; - We started before `Date`, `NaiveDateTime` and `DateTime` even existed. - That hurt. And we had **lots** of complex logic depending on time… and we also kind-of misused timezones... - Thank you core team for `{Naive,}DateTime.diff/3`

Dependency clash - Because Erlang lacked a hex.pm alternative for quite some time, some different libraries declare the same dependency name :/ - Unfortunately… there is no solution to this.

Deployment a lá "12 factor" - Deployment was a big unknown in the early days. - We had *lots* of trouble trying to mold releases into a "12 factor compliant" setup. Weird behaviours with environment variables... - We ended up running mix in production and we're doing exactly that until today. No problems since then :thumbs_up:

(and now some problems that were exclusively our fault)

Joining the "umbrella party" late - Our app started as a single phoenix app. - As the app grew large, we felt the need to split it into an umbrella application. - As you guessed, we weren't very well modularized. - Lots of pain to get this done. (It was our fault)

Fat Controllers - Is a terrible Idea. Don't do it. - See Chris' talk on Phoenix 1.3 ! https://www.youtube.com/watch?v=tMO28ar0lW8

Coarse testing - We started breaking down the fat controllers. - We got lazy and did not write much of unit tests as we were factoring out stuff from "fat controllers". Partially because things were too coupled, partially because of deadlines. - And that burned us. Bad. *Don't* ignore the testing pyramid. - In the beginning, everything was fine because the code in the controllers was concise and short. But that didn't last long.

Not using Ecto.Type - We reimplemented most of Ecto.Type functionality (very poorly) because we did not read the documentation thoroughly. - ~ code ~

Migration to a full SPA + GraphQL - We decided to reimplement the frontend as an SPA. Consuming a graphql api. - Rewriting was *not* a good idea. (it rarely is…) - **LOTS** of problems with React, ClojureScript, GraphQL, reinventing HTTP, yadda yadda. - IMHO that was our biggest mistake problem. It was not Phoenix or Elixir related at all. I will come back to this latter.

Incidental problems :s

But we also had awesome surprises!

Platform & operability! See https://www.youtube.com/watch?v=5SbWapbXhKo

Performance! (not only in production) See http://blog.plataformatec.com.br/2017/07/the-fallacies-of-web-application-performance/

Evolution! - Evolution of libraries, ecosystem, events and people interested. We started prior to phoenix & ecto 1.0; Both have come a *long* way. - Stability! Phoenix & Elixir we're beyond stable. We even forgot to add `Restart=on-failure` on systemd! - Even though the app got *really* big, it was much simpler to work with than {Rails,Spring} apps of similar size we worked with before

Contribution! - We landed a *lot* of patches in many popular libraries! (ecto, phoenix, erlcloud, plug, ja_serializer, timex, etc, etc) - We found a bug in the Elixir compiler! - We wrote some cool libraries! (like a stream-able zip encoder) - (and we also got some other stuff that we should be able to extract & open-source too)

Our hiring strategy... - It Worked! It really did! - We've got an awesome team of people (and growing!)

So, after 2 years, what about our initial concerns?

Our fears concerns - Will we find libraries & other ecosystem related stuff? Will we need to build the world? A lot before, a lot less today (always try not to!) - Will this become abandon-ware? Hell No! - How productive is this framework? Should I be settling for something like Rails instead? Phoenix is very productive! - Will this be good for "complex" business logic? Yep. No big issues - Will this be stable in production? As "stable" as "stable" can get. - (more particular to our case:) Will our hiring strategy work? It did.

Conclusion

Conclusion - Our biggest problems had nothing to do with Phoenix itself. - In 2015/2016 phoenix was enough to handle all our needs. Today it is even more so. Still lots of work to do, lots of opportunities to contribute. - Although Phoenix is new, it is *very well founded*. - The community *is* growing, the ecosystem *is* growing. Things are more and more *production ready & stable*.

If I were to build it all again, today, would I use Elixir & Phoenix? Hell YES!

Takeaway Phoenix *is* good for big "enterprise apps". The following estatement used to be very true: "you're trading maturity & productivity for performance". The amount of "maturity & productivity" we're trading is getting smaller and smaller. The future is bright. (and faster)

Obrigado!

Where to find me: Twitter: @renanranelli Github: /rranelli Blog: milhouseonsoftware.com