Elixir

Presentation introducing Elixir (and functional programming) to our team.

Transcript

1. D I S T R I B U T E D F U N C T I O N A L P R O G R A M M I N G W I T H N O R M A N T H O M A S 2 0 1 8 - 1 0 - 2 6 E L I X I R A N D O T P
2. O V E R V I E W • Current state of affairs • Functional programming • Elixir basics • OTP • Deployment • Outlook
3. W H E R E D O W E S TA N D ? C++ Java JavaScript Ruby Python Rust Go Scala OCaml Haskell Erlang Elixir Lisp Elm F# Swift Kotlin Akka Redux Smalltalk Clojure Scheme Racket Common Lisp C
4. W H E R E D O W E S TA N D ? C++ Java JavaScript Ruby Python Rust Go Scala OCaml Haskell Erlang Elixir Lisp Elm F# Swift Kotlin Akka Redux Smalltalk Clojure Scheme Racket Common Lisp C ▪Influencers
5. -JTQ
   QSPHSBNNFST )BTLFMM
   QSPHSBNNFST 7JTVBM
   #BTJD
   QSPHSBNNFST 0$BNM
   
   &SMBOH
   QSPHSBNNFST
6. S H O R T C O M I N G S O F C U R R E N T S TAT E • Mutable by default • Side-effects all over the place / unpredictable • Unexpected error? ⏩ End of program • Shared mutable state • Imperative vs. declarative • Functional Programming; What? Why? When?
 Robert C. Martin @ NDC 2014 null, None, undefined or AttributeError: 'NoneType' object has no attribute ‘width'
7. L E T ’ S P L AY A L I T T L E G A M E
8. auto result = obj.normalize();
9. obj.normalize();
10. obj.dump();
11. obj.fill(values);
12. T H E R E ’ S M O R E … get_list_status(params, arr, query_result); return_items(arr, query_result); query_result = save_ids(ids); % * actual lines from a repo
13. W H AT D O E S T H I S C O D E D O ? a = [1, 2, 3] b = a.concat(4) // a = ... ? b = ... ? b = a.push(4) // a = ... ? b = ... ? b = a.reverse() // a = ... ? b = ... ? b = a.sort() // a = ... ? b = ... ?
14. W H AT D O E S T H I S C O D E D O ? a = [1, 2, 3] b = a.concat(4) // a = [1, 2, 3]; b = [1, 2, 3, 4] b = a.push(4) // a = ... ? b = ... ? b = a.reverse() // a = ... ? b = ... ? b = a.sort() // a = ... ? b = ... ?
15. W H AT D O E S T H I S C O D E D O ? a = [1, 2, 3] b = a.concat(4) // a = [1, 2, 3]; b = [1, 2, 3, 4] b = a.push(4) // a = [1, 2, 3, 4]; b = 4 b = a.reverse() // a = ... ? b = ... ? b = a.sort() // a = ... ? b = ... ?
16. W H AT D O E S T H I S C O D E D O ? a = [1, 2, 3] b = a.concat(4) // a = [1, 2, 3]; b = [1, 2, 3, 4] b = a.push(4) // a = [1, 2, 3, 4]; b = 4 b = a.reverse() // a = [4, 3, 2, 1]; b = [4, 3, 2, 1] b = a.sort() // a = ... ? b = ... ?
17. W H AT D O E S T H I S C O D E D O ? a = [1, 2, 3] b = a.concat(4) // a = [1, 2, 3]; b = [1, 2, 3, 4] b = a.push(4) // a = [1, 2, 3, 4]; b = 4 b = a.reverse() // a = [4, 3, 2, 1]; b = [4, 3, 2, 1] b = a.sort() // a = [1, 2, 3, 4]; b = [1, 2, 3, 4]
18. “If a developer must consider the implementation of a component in order to use it, the value of encapsulation is lost.” Eric Evans *DDD guy
19. P R O S P E C T ? • Immutability as a default • Concurrency out of the box • Scalability • Resilience, high availability • Declarative problem-solving • Focus on the what, not how
20. N O TA B L E U S E R S At Discord we have been very happy using Elixir/Erlang as a core technology of our backend services. We are pleased to see additions such as GenStage that build on the rock solid technologies of Erlang/OTP. *5 million concurrent users *2 million concurrent users on a single server
21. *in Elixir F U N C T I O N A L P R O G R A M M I N G
22. P R I N C I P L E S • Immutability • No side effects • Referential transparency • Pattern matching • Lazy evaluation • No loops ➿, yes, none of the following: • for, while, do-while, repeat-until, etc. • First class functions • No shared memory between processes • yes: no locks, mutexes, semaphores etc. • yes: no threads • Communication via messages • Let it crash! • Exceptions only for unexpected, exceptional cases ⚡ * C++ Meetup “Almost no raw loops” * CppCon “Free your Functions”
 * “Structure and Interpretation of Computer Programs” J & J Sussman, 1979, MIT Press
23. O O P V S F P OOP FP paradigm everything is an object plain structs reusability through inherticance composition side effects yes no / avoided look-n-feel arr.append([4, 5, 6]) Array.append(arr, [4, 5, 6]) preference statement expression overloading through types / classes pattern matching state management in object / class instance process mindset messages to objects transformation of data concurrency manual work out of the box
24. * Scott Wlaschin: Functional Programming Patterns
25. “Object-oriented programming is an exceptionally bad idea which could only have originated in California” E. W. Dijkstra
26. “I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object-oriented language.” Tony Hoare *inventor of null
27. N O N - D E V E X A M P L E : P H O T O S H O P • FP: Adjustment layers • non-destructive transformation • don’t mutate input • composable • OO: Image adjustment & Filters • destructive, mutate original input
28. E N T E R : E L I X I R
29. T H E B A S I C S • Elixir started 2012 • compiles to Erlang byte code • Erlang from 1986/1987 • BEAM VM • dynamic, strict typing • Recursion + HOFs instead of loops • Light-weight, share-nothing concurrency • Railway-oriented programming • Lisp-like meta-programming at compile-time
 • Native support for • concurrency / async • seamless parallel processing • lazy stream-processing • Built-in • dependency management • unit tests • code formatter • remote debugging
30. • Declarative (what vs. how) • Scalability / Concurrency • no side effects = runnable in parallel • Testability / Debuggability* • functions only depend on input parameters • Simplicity • Composability / Reusability P R I N C I P L E S * yes, this word actually exists
31. T Y P E S • Atom
 :ok, :error, :timeout, :fantasy • String, Binary, Regex
 “some string”, <<126, 127>>, ~r/^.*$/ • Integer • Float • Date, DateTime, Time, Date.Range • PID • Function • Exception • Stream • Collections • Tuple {1, 2, 3} • List, Keyword List [1, 2, 3], [k1: 1, k2: 2] • Map %{ “key” => “value” } • MapSet • Struct • Record all are deeply immutable
32. | >
33. P I P E S A R E FA M I L I A R > du -h -d 0 node_modules/* | grep "M\s" | gsed 's/\s*\([0-9]\+\(\.[0-9]\+\)\?\)M\s\+\(.\+\)/\1 MB\t\3/' | sort -n | tail 5.1 MB node_modules/fbjs 6.3 MB node_modules/@babel 6.4 MB node_modules/less 7.3 MB node_modules/core-js 9.4 MB node_modules/faker 13 MB node_modules/@ant-design 16 MB node_modules/rxjs 51 MB node_modules/@material-ui 52 MB node_modules/antd 201 MB node_modules/puppeteer > awk '$1 !~ /(EBOOK|BOOK)/{ print $1 }' dotbooks_sales.csv | sort | uniq -c 59724 ebook 963 epub 1 product > awk '$4 == "EUR" { print $2, $3 }' dotbooks_sales.csv | tail -n +2 | awk '{q+=$1; amount+=$2} END {print q, amount}’ 2163869 4.84478e+06
34. "palindrome" |> String.reverse
35. [1, 2, 3] |> Enum.map(fn x -> x*x end)
36. P I P E S MySQL mysql = ... Browser browser = ... html = browser.visit(page) print(html) data = scrape(html) mysql.store_to_db(data) browser |> visit(page) |> IO.inspect # easy debugging / logging |> scrape(:metrics) |> MySQL.store_to_db()
37. T R A N S F O R M AT I O N A P P L I C AT I O N Task: find the first n prime numbers in a range from 2 to 1000 bool is_prime(const int num) { ... } std::vector<int> result; for (int i = 2; i < 1000 && result.size() < n; i++) { if (is_prime(i)) result.push_back(i); } is_prime = fn num -> ... end 2..1000 |> Stream.filter(&is_prime/1) |> Enum.take(n) const is_prime = num => { ... } result = [...Array(1000).keys()] .filter(is_prime).slice(0, n) ≠
38. T R A N S F O R M AT I O N A P P L I C AT I O N Task: find the first n prime numbers in a range from 2 to 1000 bool is_prime(const int num) { ... } std::vector<int> result; for (int i = 2; i < 1000 && result.size() < n; i++) { if (is_prime(i)) result.push_back(i); } is_prime = fn num -> ... end 2..1000 |> Stream.filter(&is_prime/1) |> Enum.take(n) is_prime = fn num -> ... end 1..1000 |> Flow.from_enumerable |> Flow.filter(&is_prime/1) |> Enum.take(n)
39. A N O T H E R E X A M P L E std::vector<Vlb_Kat> search_vlb_kats(const size_t limit) const { IDVector all_vlb_kat_ids; SearchIDs<Vlb_Kat>::search(all_vlb_kat_ids); NGString query = ctx_.url().parameters().get_parameter_single("q").lowercase(); std::vector<Vlb_Kat> vlb_kats; size_t matches = 0; for(auto const vlb_kat_id : all_vlb_kat_ids) { if(matches >= limit) break; Vlb_Kat vlb_kat; if(vlb_kat.load(vlb_kat_id, LoadContextCache()) && vlb_kat.get_bod_supported() == Vlb_Kat::BOD_SUPPORTED::YES) { if(NGString::npos != vlb_kat.get_description().lowercase().find(query)) { vlb_kats.push_back(vlb_kat); ++matches; } } } return vlb_kats; } defmodule Request.RPC.AutoSearch do def search_vlb_kat(request_params, limit) do query = request_params["q"] |> String.downcase() DB.search("vlb_kat") |> Stream.map(&load_vlb_kat/1) |> Stream.filter(fn vlb_kat -> vlb_kat.bod_supported end) |> Stream.filter(fn vlb_kat -> vlb_kat.description |> String.downcase() |> String.contains?(query) end) |> Enum.take(limit) end end
41. E L I X I R V S C + + // lambda auto println = [](const char *message){ std::cout << message << std::endl; }; # lambda println = fn message -> IO.puts message end // fold expressions (C++17) bool all(bool... args) { return (... && args); } # Enum.all?([true, true, false, true]) // “filter” std::copy_if(x.begin(), x.end(), std::back_inserter(y), predicate); x |> Enum.filter &predicate/1 Enum.filter(x, &predicate/1)
42. E L I X I R V S P Y T H O N config = { "host": "api.op.com", "api_key": "1_1R_3" } create_url = lambda: f"https://{config['host']}/rpc/me?access_token={config['api_key']}" config.update({ "api_key": "1_4R_9" }) url = create_url() # https://api.op.com/rpc/me?access_token=1_4R_9 config = %{ host: "api.op.com", api_key: "1_1R_3" } create_url = fn -> "https://#{config.host}/rpc/me?access_token=#{config.api_key}" end config = Map.put(config, :api_key, "1_4R_9") url = create_url.() # https://api.op.com/rpc/me?access_token=1_1R_3
43. E L I X I R V S P Y T H O N Python Elixir 1 = 1 SyntaxError: can't assign to literal 1 a, b = 1, 2 a = 1 b = 2 ** (SyntaxError) iex:1: syntax error before: ',' [1,2,3] == [1,2,3] True true “foo” in [“foo”, “bar”] True true “foo” == “foo” True true not not “” / !!”” False true "" or False / "" || false False “” 0 or True / 0 || true True 0 1 == true True false 2 == true False false true + true 2 ArithmeticError
44. E L I X I R V S E S 6 / 7 / 8 var people = [ { name: 'Alice', age: 21 }, { name: 'Max', age: 20 }, { name: 'Jane', age: 20 } ]; // conventional function groupBy(objectArray, property) { return objectArray.reduce(function (acc, obj) { var key = obj[property]; if (!acc[key]) { acc[key] = []; } acc[key].push(obj); return acc; }, {}); } // lodash _.groupBy(people, p => p.age) people = [ %{name: "Alice", age: 21}, %{name: "Max", age: 20}, %{name: "Jane", age: 20} ] people |> Enum.group_by fn item -> item.age end ES6 Elixir 1 + “1” + 1 “111” ArithmeticError “1” - 0 1 ArithmeticError {} + [] 0 ArithmeticError [1,2,3] == [1,2,3] false true [1,2,3] === [1,2,3] false true “foo” in [“foo”, “bar”] false [“foo”, “bar”] .includes("foo") true “foo” === “foo” true true !!”” false true true + true 2 ArithmeticError
45. E L I X I R V S E S 6 / 7 / 8 ( C O N T D ) • dependency hell thomas@thomas:~/git/trunk/react (stable•) $ grep "^\s\{1,4\}\"[a-zA-Z0-9_-.]\+\": {" package-lock.json | wc -l 935 thomas@thomas:~/git/trunk/react (stable•) $ du -h -d0 node_modules 761M node_modules added 413 packages from 135 contributors, removed 98 packages, updated 436 packages and audited 7315 packages in 64.839s found 406 vulnerabilities (339 low, 42 moderate, 20 high, 5 critical) run `npm audit fix` to fix them, or `npm audit` for details • immutablejs: 139 KB • lodash: 115 KB • redux: 3.5 MB • flow: 5.6 MB • webpack to tame the beast
46. E L M ? hmmm… -- CARD VIEW taskItemView : Int -> Task -> Html Msg taskItemView index task = li [ class "task-item", attribute "draggable" "true", onDragStart <| Move task, attribute "ondragstart" "event.dataTransfer.setData('text/plain', '')" ] [ text task.name , button [ class "btn-delete" , onClick <| Delete task.name ][ text "+" ] ] -- COLUMN VIEW taskColumnView : String -> List Task -> Html Msg taskColumnView status list = div [ class <| "category " ++ String.toLower status, attribute "ondragover" "return false", onDrop <| DropTask status ] [ h2 [] [ text status ], span [] [ text (toString (List.length list) ++ " item(s)") ], ul [] (List.indexedMap taskItemView list) ]
47. I M I TAT I O N S I N T H E W I L D Python Javascript C++ Functional Programming OSlash / toolz / pyfunctional / … ramda / Sanctuary FunctionalPlus Immutability ⛔ / pyrsistent ⛔ / immutable.js ✅ Pipe Operator ⛔ this, this, this and this… ⏳ official proposal ⛔ imitation Pattern Matching ⛔ decorators or outdated module ✅ ⛔ Mach7 or Patterns Monads ⛔ OSlash / PyMonad ⛔ MonetJS ✅ std::optional / neither Functors ✅ ✅ ✅ map / filter / reduce ✅ ✅ ✅ std::transform / std::copy_if… Currying / Partial Application manual / ✅ manual or ramda / ✅ ⛔ via templates Tail Call Optimization ⛔ ⛔ ⛔
48. PAT T E R N M ATC H I N G
49. PAT T E R N M AT C H I N G def fac(1), do: 1 def fac(n), do: n * fac n - 1 [1, a, 3] = [1, 2, 3] [a | _] = [1, 2, 3] [_ | rest] = [1, 2, 3] [a, b | _] = [1, 2, 3] [a | [b | _]] = [1, 2, 3] %{config: %{destination: {"https", host, 443}}} = %{config: %{destination: {"https", "api.op.com", 443}}} %{config: %{destination: {"http", host, 80 }}} = %{config: %{destination: {"https", "api.op.com", 443}}} %{config: %{destination: {_, host, _ }}} = %{config: %{destination: {"https", "api.op.com", 443}}}
50. R E C U R S I O N def deep_get(%{} = m, [h | t]) do m |> Map.fetch!(h) |> deep_get(t) end def deep_get([_ | _] = m, [h | t]) when is_integer(h) do m |> Enum.fetch!(h) |> deep_get(t) end def deep_get([_ | _] = m, [h | t]) when is_atom(h) do m |> Keyword.fetch!(h) |> deep_get(t) end def deep_get(m, []) do m end def fun do map = %{ "user" => %{ "id" => 123, "name" => %{ "first" => “Siglinde", "last" => “Petersen" }, "documents" => [123, 456, 789] } } first_name = deep_get(map, ["user", "name", "first"]) first_document = deep_get(map, ["user", "documents", 0]) {first_name, first_document} end
51. S O , W H Y N O T P L A I N E R L A N G T H O U G H ? ⚠ It has no pipe operator ⚠ !Es hat keinen Pipe-Operator! !Não tem operador de tubulação! !Вона не має трубного оператора! !ሏ຅༗؅ಓࢉࢠ!
52. D R A W B A C K S • Not well suited for heavy, mathematical operations • Not much machine learning
53. O T P O P E N T E L E C O M P L AT F O R M
54. E R L A N G H E R I TA G E • Robust • Fault tolerant • Scalable • Distributed • since the 1980s
55. P R I N C I P L E S • Processes, processes everywhere • Processes communicate via messages • Implementation of patterns revolving around process lifecycle and management • Every process is a supervisor or worker • Application is a tree of processes • Adaptable generic solution for specific problem send pid, message receive do message -> do_something(message) end spawn fn -> IO.puts “Heavy duty!” end
56. A C T O R M O D E L
57. P R O C E S S L I N K S iex> spawn_monitor fn -> IO.puts "Hello from #{inspect Node.self}" end Hello from :miranda@io {#PID<0.118.0>, #Reference<0.3297442424.34078724.196482>} iex> flush {:DOWN, #Reference<0.3297442424.34078724.196482>, :process, #PID<0.118.0>, :normal} • spawn
 • spawn_link
 • spawn_monitor
58. P E R F O R M A N C E • Elixir/Erlang processes are cheap • 1µs to spawn—yes, µs, not ms—for up to 3000 procs • can handle over 100k procs on a single machine • 0.8µs for sending a message for up to 30k procs
59. G E N E R I C V S S P E C I F I C • Problem may be domain specific, technical challenge most likely not • Reinvention addiction • challenging • fun • waste of time
60. O T P A P P L I C AT I O N • Similar to a service within micro-services, often smaller • Functionality can be started/stopped as a unit • think: systemd units • Reusable, isolated components
61. S U P E R V I S O R S • Starts / stops / monitors other processes • Decides what happens if a sub-process fails • Has restart strategies • No other responsibilities
62. M I N I D E M O
63. W O R K E R S • Do the actual work • May or may not be supervised • Can crash • may or may not be restored after a crash • Are managed by supervisor
64. O T P PAT T E R N S / T O O L S • Battle-tested collection of
 generic implementations: • GenServer • GenTCP • GenUDP • GenEvent • GenStatem • GenStage* • Tasks • (D)ETS • Mnesia
65. G E N S E R V E R • Generic Server • Provides • call (synchronous) • cast (asynchronous) • info • can spawn new process for each request • can provide worker pool • can do load balancing
66. G E N S TA G E P R O D U C E R P R O D U C E R C O N S U M E R P R O D U C E R C O N S U M E R P R O D U C E R C O N S U M E R C O N S U M E R • Pipeline
 • Back-pressure
 • Consumer pulls data
68. E T S / D E T S / M N E S I A • ETS = Erlang Term Storage • in-memory ephemeral storage (key-value / cache) • single-node • DETS = Disk-based ETS • Mnesia • powerful, distributed relational database • multi-node
69. O T P R E L E A S E S • Builds self-contained package • Highly configurable • Versioning • Hot up/downgrades • code_change event/callback • Distributed applications • Rollback possible
70. D E P L O Y M E N T • Declare settings for dev, test and prod • Building and deploying via Docker possible • tar ball is generated • copy tar ball to target machine
71. D I ST R I B U T I O N
72. D I S T R I B U T I O N • Support for distribution / clustering from day 1 • Processes can be run anywhere, on any machine • Multiple nodes can be connected • Support for load balancing • Support for “leader”, fail-over, take-over across nodes • Ability to capture state when process crashes
73. L E T ’ S S E T U P A C L U S T E R Node.connect(:"name@host.tld")
74. L E T ’ S T RY I T > iex --sname n1 --cookie mysecret123 iex> Node.list iex> Node.connect :n2@io iex> Process.register self(), :carla iex> send {:anastacia, :n2@io}, {"Yo, Anastacia!", self()} iex> flush > iex --sname n2 --cookie mysecret123 iex> Node.list iex> Node.list iex> Process.register self(), :anastacia iex> receive do {msg, from} -> IO.puts "You've got mail! from #{inspect from}: ‘#{msg}'" :timer.sleep(3_000) send from, “back to you!” end iex> flush * possible to register global names
75. D I S T R I B U T I O N P O W E R E D B Y E L I X I R • Respond to demand • DynamicSupervisor • set bounds, i.e. min/max workers • Distribute workload across nodes • Automated, reactive cluster formation, discovery and healing
 via swarm / libcluster • Asynchronicity* permits easier monitoring / introspection of application state • Encryption using TLS * tongue twister
76. S T I L L • Even with Erlang/Elixir,
 distributed systems don’t fall from the sky • But the Erlang ecosystem provides powerful tools
 to manage the complexity, and to focus on the actual task at hand
77. OT H E R CA N DY
78. Phoenix web framework Ecto database binding Nerves embedded Elixir Absinthe GraphQL library
79. T O O L I N G :observer monitoring mix formatter code formatter dialyzer code analyzer credo linter inch documentation coverage IEx REPL + debugging ExUnit testing PropEr / PropCheck property testing benchee benchmark
82. W H E R E TO G O F R O M H E R E ?
83. ⬆ ask for help ⬆
84. R E S O U R C E S • Websites • Elixir Lang • HexDocs • Elixir School • Online Courses • Developing With Elixir/OTP
 • Literature • Introducing Elixir (2nd edition) • Programming Elixir 1.6 • Mastering Elixir • Designing for Scalability with Erlang/OTP • Youtube channels • ElixirConf • CodeSync • Erlang Solutions
85. D I ST R I B U T E A L L T H E T H I N G S