POLYGLOTTIN’ FORᐧTHEᐧWIN /bentanweihao /benjamintanweihao [email protected] by benjamin tan wei hao Hello! My name is Benjamin Tan.

and I work for Neo.

This is Tim O’Reilly, the same guy behind

the O’Reilly books. ! He once gave a talk he gave titled “Watching the Alpha Geeks”.

We look for people who appear to be doing magic, and ask them how they do it. In it, he said, and I quote, “This is how we get most of our good ideas at O’Reilly. ! We look for people who appear to be doing magic, and ask them how they do it.

There are always people in any field who are the most clued in to the deep trends, who seem to be playing with all the coolest stuff, and seem to have their finger in everything before most people even know about it.” ! There are always people in any field who are the most clued in to the deep trends, who seem to be playing with all the coolest stuff, and seem to have their finger in everything before most people even know about it.” ! This was what I was observing with Elixir, a new programming language that I was hearing a lot about. !

José Valim @josevalim This is José Valim. You may know him from his contributions to Rails and gems such as Devise.

Nowadays, he is also known as the creator of Elixir, a new programming language.

A couple of months ago, Jose did a Peepcode screencast.

Not long after, the Pragmatic Bookshelf, O’Reilly, and Manning all announced Elixir books. !

Dave Thomas has been a very early supporter and advocate Ruby and Rails. ! He is also often credited as being one of the the people responsible for bringing Ruby to the English speaking world with the PickAxe book. ! Nowadays, he is doing the same thing with Elixir.

– Joe Armstrong, Erlang inventor This is Joe Armstrong, one of the inventors of Erlang. ! He wrote about his experience with Elixir in a blog post.

“This is Good Shit.” – Joe Armstrong, Erlang inventor Suffice to say, he had generally positive reviews about it. ! What the heck was going on? The language hasn’t even hit 1.0 yet all this content was being created, and quite a few smart people really liked the language. ! So I decided to investigate. ! The more I played with Elixir, the more I liked it. I was hooked. !

polyglot (Someone who knows and can use multiple languages) A polyglot is someone who knows and can use multiple languages.

polyglot Programming Similarly, a polyglot programmer knows and can use multiple programming languages. ! !

This talk is about looking beyond Ruby. ! I won't try to convert you away from Ruby. ! But I might just persuade you to add Elixir to your toolset.

So what is Elixir and what exactly makes it special? !

Elixir is a

functional

meta-programming aware language built on top

of the Erlang VM. It focuses on

tooling to leverage Erlang's abilities to build concurrent, distributed and fault-tolerant applications.

Concurrency- Oriented One of the reasons I was attracted to Elixir is because it is a Concurrency–Oriented language.

A Process <0.81.0> The main concurrency primitive in Elixir in the process. ! Processes in Elixir are not the same as the ones in the OS. ! They are completely managed by the Erlang Virtual Machine.

Erlang Virtual Machine x 13 million processes! ! Elixir programmers create processes just as Ruby programmers create objects. In fact, a single Erlang VM can support up to 13 million processes.

Erlang Virtual Machine Elixir processes can run across all CPUs with very little overhead. ! Here we have a Uniprocessor machine running a single Erlang VM.

Erlang Virtual Machine If we have multi-processor hardware, the runtime system automatically distributes the workload over the available CPU resources without any changes to the program.

<0.81.0> <0.83.0> <0.82.0> send <0.82.0>, “Ohai!” send <0.83.0>, “Kthnxbai!” The only way a processes communicate with each other is through sending and receiving messages. As long as you know the process id, you can send the process a message. ! This is an important point. ! A process A cannot modify the memory of process B directly. The only way to potentially change the state of a process is to send a message to it.

http://youtu.be/I3twOGVC0lU Video at http://youtu.be/I3twOGVC0lU Messages are sent asynchronously, much like fire and forget. We will see this in action soon.

Ackermann Function … This is the Ackermann function. ! This function takes in 2 arguments m and n, of which both must be 0 or more. ! The first 2 cases are pretty straightforward. It is the third case that gets interesting. Notice that the second argument in the 3rd case calls itself. This makes the computation grow pretty quickly, and therefore makes for a fun example.

Ackermann Function … defmodule  Ack  do      def  ack(0,  n),  do:  n  +  1      def  ack(m,  0)  when  m  >  0,  do:  ack(m-­‐1,  1)      def  ack(m,  n)  when  m  >  0  and  n  >  0  do            ack(m-­‐1,  ack(m,n-­‐1))      end   end This is the Ackermann function in Elixir. ! Elixir programs are organised in modules. Within modules, are function clauses.

defmodule  Ack  do      def  ack(0,  n),  do:  n  +  1      def  ack(m,  0)  when  m  >  0,  do:  ack(m-­‐1,  1)      def  ack(m,  n)  when  m  >  0  and  n  >  0  do          ack(m-­‐1,  ack(m,n-­‐1))      end   !    def  loop  do          receive  do              {m,  n}  -­‐>                  IO.puts  ack(m,  n)              msg  -­‐>                  IO.puts  “#{self}  received:  #{msg}“          end          loop      end   end This is the full Ackermann program, with an additional loop function. ! This loop function, when executed in a process, enables the process to receive and respond to messages. ! Let’s see how we can do that. !

Step 1: Spawn a Process w1  =  spawn(Ack,  :loop,  []) <0.82.0> def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   To create a process, we use the spawn function. ! The spawn function takes in 3 arguments: The module name, the function name, and the arguments to the function. ! Since loop takes in 0 arguments, we pass in an empty list. ! The return value of the spawn function is a process id, or pid for short. It is used to reference processes. ! Now, our process is ready to receive messages.

Step 2: Sending a Message w1  =  spawn(Ack,  :loop,  []) <0.82.0> send(w1,  {1,  2}) {1,  2} def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   To send a message to a process, we use the built in send function. ! This takes in 2 arguments, the pid, and the message.

Step 2: Sending a Message w1  =  spawn(Ack,  :loop,  []) <0.82.0> send(w1,  {1,  2}) {1,  2} def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   ! The kinds of messages that a process can handle are declared in the receive block. ! If the pattern of the message matches those of patterns defined in the receive block, then the body is executed. !

Step 2: Sending a Message w1  =  spawn(Ack,  :loop,  []) <0.82.0> send(w1,  “LOLWUT”) “LOLWUT” def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   If a message doesn’t match the first pattern, the next patten will be tried, and so on.

Step 2: Sending a Message w1  =  spawn(Ack,  :loop,  []) <0.82.0> send(w1,  {1,  2}) {1,  2} def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   “Result:  4” In this example, we send over a 2 element tuple. ! That matches the first pattern and the result is computed. ! !

Step 2: Sending a Message w1  =  spawn(Ack,  :loop,  []) <0.82.0> send(w1,  {1,  2}) {1,  2} def  loop  do      receive  do          {m,  n}  -­‐>              IO.puts  ack(m,  n)          msg  -­‐>              …      end      loop   end   “Result:  4” When the computation is done, the loop function calls itself again with a tail recursive call. ! In Elixir, looping is done via recursion, and is a very common pattern in Elixir programs. ! This is needed is because we want the process to be able to receive multiple messages. ! Without the loop, the process will exit after receiving a single message. ! Let’s see a quick demo.

Video at: http://youtu.be/dS7pBeBz258 Video at: http://youtu.be/dS7pBeBz258 ! Here, I want to show you that we can run the functions directly. That is, we are not running this functions in a process. ! [Play] ! For simple computations, that is probably fine. But the moment you run more computationally intensive operations, you will block the caller. ! In this case, the shell session is blocked, and will only be unblocked once the computation is completed.

Video at: http://youtu.be/LwPhCiECcZQ Video at: http://youtu.be/LwPhCiECcZQ ! Things get a little more fun now. ! We will create a process with spawn. We will send the process the exact same message as before. Since we’re running a process, it will not block the caller, which is the shell, is not blocked. ! [Play] ! Notice for simple jobs, the results return almost immediately. For more complex jobs, only the message is returned. You will have to wait a while more for the result to appear. ! But unlike the previous case, the calling process, in this case the shell session, is not blocked in any way. !

Video at: http://youtu.be/hzt1SOAHxFU Video at: http://youtu.be/hzt1SOAHxFU ! [Play] ! When we send the process a message other than a 2 element tuple, we fall back to the second pattern that matches anything.

Video at: http://youtu.be/-DvXH4GzrQ0 Video at: http://youtu.be/-DvXH4GzrQ0 ! Let’s take things up a notch. We are going to start 4 processes, and give each of them a computationally intensive job to handle. ! [Play] ! See how the activity monitor light up completely. ! We can go ahead and run another process and give it a job to handle. ! On the other hand, if we send messages to a busy process, it will simply be buffered, to be processed later once the current computation is completed.

Fault-Tolerant Elixir has also baked in features that make it easy to build fault tolerant systems. ! Fault-tolerance in this context means making the system stay up despite failure

Let it Crash! Let it Crash is part of the Erlang philosophy. In general, defensive programming is frowned upon. ! You are going to screw up eventually, and your code will have bugs. ! Instead of trying to figure out all the corner cases, have the process to fail quickly, and let another process handle that failure.

Supervisor Worker One of the fault tolerance features of Elixir comes in the form of supervisors.. ! The Supervisor’s only job is to monitor child processes.

Supervisor Worker Supervisors can also be supervised. ! This means that supervision trees can be layered to form and even bigger supervision tree.

<0.81.0> <0.82.0> When a child process dies, the supervisor can react in a number of ways. ! For example, it can simply restart the failed child. !

Or it can terminate all the child processes under the tree, and restart them.

Video at: http://youtu.be/VK9ECKhdWFI Video at: http://youtu.be/VK9ECKhdWFI ! I’m going to show a demo where you can see how a supervisor restarts a killed child process. ! [Play] ! Red balls are unnamed processes, and Blue balls are named processes. The green stuff you can ignore. ! Overall, these are the processes present each time you start iex. ! Now, I’m going to start a bunch of child processes on one of the supervisors. ! Then, I’m going to forcefully kill all the children one by one. See what happens next.

Distribution https://github.com/benjamintanweihao/primy The next thing I want to showcase is Distributed Elixir, which I think is one of the most fun features the Erlang Virtual machine gives us. ! The Erlang Virtual Machine makes connecting nodes to form a cluster pretty easily. !

Erlang Runtime System Erlang Runtime System Network A B C send(B,  Msg) Processes in a cluster are location transparent. This means that message passing between processes in the same computer ..

Erlang Runtime System Erlang Runtime System Network A B C send(C,  Msg) is just as easy between processes on a different node as you know the process id.

We have hear a 5 node cluster consisting of a server node, and 4 other worker nodes. ! The purpose of this distributed system that I’m about to demo is to check for prime numbers.

Here’s an example of the messages that gets passed between a worker process and a server. ! The worker requests for a number to test. The server responds with a candidate number. ! When the worker receives the number, it goes on and checks for primality. ! Once done, it sends the server back the reply. Finally, it requests for another number to test. And the cycle repeats.

Note that this back and forth message passing can happen with multiple processes spanning multiple nodes. ! The beauty of this is the server and worker nodes do not have to be on the same computer. ! Let’s see a demo.

Video at: http://youtu.be/NYts9xfnCIw Video at: http://youtu.be/NYts9xfnCIw ! In this example, I’m going to set up a 3 node cluster. ! As long as you know the name and the IP address, setting up a cluster is a pretty trivial affair. ! [Play]

Video at: http://youtu.be/_5GrGY6NzNM Video at: http://youtu.be/_5GrGY6NzNM ! Next, I’m going to run the server on the top left node, and worker processes on the remaining nodes. ! [Play] ! Then, I’ll go ahead and create more worker processes.

Video at: http://youtu.be/G-LPttJOjfc Video at: http://youtu.be/G-LPttJOjfc ! Finally, I’m going to connect a Raspberry Pi to the cluster. It then takes part in the discovery of prime numbers. !

The Pipe Operator |> I want to introduce my all time favourite programming operator – The pipe. ! José stole the pipe operator from F#. What it does is to pipe the value on the left side as the first argument to the function call on the right side, much like the Unix pipe. ! This operator alone changes the way you think about programming. ! This operator encourages you to think of programming as transformation of data via multiple functions. ! Let me show you an example.

    Enum.reverse(Enum.map(1..10,  fn  x  -­‐>  x  *  2  end))   [20,  18,  16,  14,  12,  10,  8,  6,  4,  2] ! Here is one way to write this function. ! Here, we are taking a sequence of numbers from 1 to 10, and then multiplying each of the numbers by 2. Finally, we reverse the result. ! Notice how you have to read from inside out to understand what the function does. ! Now check this out.

1..10     |>  Enum.map(fn  x  -­‐>  x  *  2  end)     |>  Enum.reverse     Enum.reverse(Enum.map(1..10,  fn  x  -­‐>  x  *  2  end))   Here’s the same function, but expressed using the pipe operator. ! We pipe the sequence into the map function, and the result form the map function is piped into reverse. ! Data transformation in a series of functions. ! That looks way prettier to me.

Rails + &Elixir WORKERS So now, we’ve come to the last bit of the talk. ! Hopefully, I’ve managed to convince you that Elixir is pretty awesome. ! We are going to see how we can use Elixir in our Ruby projects. ! Here, I’m going to use Rails that has Sidekiq installed. ! For those not familiar with Sidekiq, it describes itself as being a simple and efficient background job processor for Ruby.

How does work? Before we start messing around with Sidekiq, we need to understand a little what goes on under the hood. ! Here’s a slightly simplified version of how Sidekiq works.

Step 1: Define the Worker class  HardWorker      include  Sidekiq::Worker          def  perform(user_name)          #  Do  hard  work      end   end The first step is to write the worker.

Step 1: Define the Worker class  HardWorker      include  Sidekiq::Worker          def  perform(user_name)          #  Do  hard  work      end   end We need to include the Sidekiq Worker module.

Step 1: Define the Worker class  HardWorker      include  Sidekiq::Worker          def  perform(user_name)          #  Do  hard  work      end   end Then, we need to implement the perform method.

Step 2: Invoke the Worker HardWorker.perform_async("bentanweihao") To call the worker, we execute the perform async method with the appropriate arguments.

Step 3: A job is created and goes into the Redis queue {        :jid                  =>  SecureRandom.hex(12)      :queue              =>  "queue:default",      :class              =>  "HardWorker",      :args                =>  ["bentanweihao"],      :enqueued_at  =>  Time.now.to_f   }.to_json Job is pushed to the left of the queue The Sidekiq client then inserts a JSON hash containing information such as the job id, name of the queue, the class and it’s arguments. ! These are the information needed to execute the worker code in a thread.

Step 4: Job gets popped by the Poller Job is popped by the poller {        :jid                  =>  SecureRandom.hex(12)      :queue              =>  "queue:default",      :class              =>  "HardWorker",      :args                =>  ["bentanweihao"],      :enqueued_at  =>  Time.now.to_f   }.to_json Sidekiq has a poller that polls the a Redis queue for any new jobs. ! If there is one, it is popped of the queue by the poller.

#  This  is  executed  in  a  thread   HardWorker.new.perform("bentanweihao") Step 4: Thread created to run Worker {        :jid                  =>  SecureRandom.hex(12)      :queue              =>  "queue:default",      :class              =>  "HardWorker",      :args                =>  ["bentanweihao"],      :enqueued_at  =>  Time.now.to_f   }.to_json Finally, Sidekiq examines the hash and executes the appropriate worker in a thread

You can write workers in any language! But here’s the thing. ! You can write the workers in any language you want, as long as you have a Redis client.

You can write workers in any language! But let’s see how we can use Elixir to write Sidekiq workers.

Step 1: Define the Worker class  HardWorker      include  Sidekiq::Worker          def  self.perform_async(*args)        json  =            {                queue:  "queue:elixir",              class:  "HardWorker",                args:  args,                  jid:  SecureRandom.hex(12),   enqueued_at:  Time.now.to_f          }.to_json                    client  =  Sidekiq.redis  {  |conn|  conn  }          client.lpush(queue,  json)      end      #  ...   end Here, we override perform_async that is previously defined in Sidekiq::Worker. ! !

Step 1: Define the Worker  def  self.perform_async(*args)        json  =            {                queue:  "queue:elixir",              class:  "HardWorker",                args:  args,                  jid:  SecureRandom.hex(12),   enqueued_at:  Time.now.to_f          }.to_json                    client  =  Sidekiq.redis  {  |conn|  conn  }          client.lpush(queue,  json)      end The trick is to insert the JSON hash into another queue that the Sidekiq poller will not look for. ! In this example, we are inserting it into “queue:elixir” instead of “queue:default”.
 ! Other than that, the JSON remains the same. !

Step 1: Define the Worker  def  self.perform_async(*args)        json  =            {                queue:  "queue:elixir",              class:  "HardWorker",                args:  args,                  jid:  SecureRandom.hex(12),   enqueued_at:  Time.now.to_f          }.to_json                    client  =  Sidekiq.redis  {  |conn|  conn  }          client.lpush(queue,  json)      end We then manually push the JSON into the Redis queue, again this time, to the queue that Sidekiq is not looking out for.

Step 2: Invoke the Worker HardWorker.perform_async("bentanweihao") We invoke the worker as per normal.

Step 3: Job gets popped by the Elixir Poller {        :jid                  =>  SecureRandom.hex(12)      :queue              =>  "queue:default",      :class              =>  "HardWorker",      :args                =>  ["bentanweihao"],      :enqueued_at  =>  Time.now.to_f   }.to_json Job is popped by the poller Now, we have an Elixir poller that is looking out for jobs that are being pushed into queue:elixir.

Step 4: Job gets popped by the Elixir Poller defmodule  Poller  do      use  GenServer      #  ...      def  poll(redis_client)  do          IO.puts  "Polling  ..."            redis_client            |>  Exredis.query(["RPOP",  "queue:elixir"])          |>  WorkerSupervisor.new_job   !        :timer.sleep(10)          poll(redis_client)      end   end Here is roughly how the Elixir poller looks like. ! The function poll takes in the process id of the Redis client, then executes the right pop query in the elixir queue. ! It then hands over the job to the worker supervisor who will then delegate it to the worker. It then sleeps for a while, and repeats the entire process with a tail recursive call to itself.

Step 6: Job handled by Elixir Worker defmodule  WorkerSupervisor  do      use  Supervisor      #  ...        def  new_job(job)  do          {:ok,  pid}  =  :supervisor.start_child(__MODULE__,  [])          Worker.run(pid,  job)      end   end The supervisor then receives a job from the poller. !

Step 6: Job handled by Elixir Worker defmodule  WorkerSupervisor  do      use  Supervisor      #  ...        def  new_job(job)  do          {:ok,  pid}  =  :supervisor.start_child(__MODULE__,  [])          Worker.run(pid,  job)      end   end ! The supervisor then creates a new supervised worker,

Step 6: Job handled by Elixir Worker defmodule  WorkerSupervisor  do      use  Supervisor      #  ...        def  new_job(job)  do          {:ok,  pid}  =  :supervisor.start_child(__MODULE__,  [])          Worker.run(pid,  job)      end   end ! and hands the job to it.

defmodule  Worker  do      use  GenServer          def  run(pid,  job)  do          :gen_server.cast(pid,  {:run,  job})      end          def  handle_cast({:run,  job},  client)  do          #  Computed  result  from  Elixir  worker          args  =  [process_stats(job[“args"]  |>  Enum.first)]          job    =  JSON.decode!(job)              new_job  =  HashDict.new                                |>  HashDict.put(:queue,  "queue:default")    #  Use  the  Rails  Sidekiq  queue                              |>  HashDict.put(:jid,  job["jid"])                              |>  HashDict.put(:class,  "HardWorker")          #  Matches  Rails  Worker                              |>  HashDict.put(:args,  args)                              |>  HashDict.put(:enqueued_at,  job["enqueued_at"])                              |>  JSON.encode!              client  |>  Exredis.query(["LPUSH",  queue,  new_job])                    {:noreply,  client}      end   end This is roughly how the worker looks like. ! Again, the implementation is not important. ! The important thing is to see what the worker does to the JSON hash. Let’s take a closer look.

!        job    =  JSON.decode!(job)              new_job  =  HashDict.new                                |>  HashDict.put(:queue,  "queue:default")                                |>  HashDict.put(:jid,  job["jid"])                              |>  HashDict.put(:class,  "HardWorker")                                      |>  HashDict.put(:args,  [process_stats(job["args"])])                              |>  HashDict.put(:enqueued_at,  job["enqueued_at"])                              |>  JSON.encode!              client  |>  Exredis.query(["LPUSH",  queue,  new_job])   Now, this is where we perform a sleight of hand. ! Here, we modify 2 things. ! First, we modify the queue so that it matches the queue that the Sidekiq poller looks out for. !

!        job    =  JSON.decode!(job)              new_job  =  HashDict.new                                |>  HashDict.put(:queue,  "queue:default")                                |>  HashDict.put(:jid,  job["jid"])                              |>  HashDict.put(:class,  "HardWorker")                                      |>  HashDict.put(:args,  [process_stats(job["args"])])                              |>  HashDict.put(:enqueued_at,  job["enqueued_at"])                              |>  JSON.encode!              client  |>  Exredis.query(["LPUSH",  queue,  new_job])   More importantly, the computed result by the Elixir worker is stored as the arguments. !

!        job    =  JSON.decode!(job)              new_job  =  HashDict.new                                |>  HashDict.put(:queue,  "queue:default")                                |>  HashDict.put(:jid,  job["jid"])                              |>  HashDict.put(:class,  "HardWorker")                                      |>  HashDict.put(:args,  [process_stats(job["args"])])                              |>  HashDict.put(:enqueued_at,  job["enqueued_at"])                              |>  JSON.encode!              client  |>  Exredis.query(["LPUSH",  queue,  new_job])   Finally, the JSON is inserted into the Sidekiq queue. !

Step 7: Profit! class  HardWorker      include  Sidekiq::Worker          def  perform(*args)          #  args  is  the  result  from            #  the  Elixir  worker      end   end The Sidekiq poller will pick up the job that we have just inserted. ! Then it will call the HardWorker class, with the arguments that has been previously computed by the Elixir worker. ! So much for looking at code. Let me show you a quick demo.

http://youtu.be/voLymbJ932M Video at: http://youtu.be/voLymbJ932M ! In this demo, I have 3 split screens. ! I’ll start a Rails console at the top left, run sidekiq at the bottom left, and finally the elixir poller on the right hand side. ! [PLAY] ! From the Rails console, I’m going to add a job, exactly just like how you would normally do in your Rails controllers. ! All the Elixir worker is going to do is take the arguments which are Strings, up case the string, then reverse it. ! Once the Elixir worker is done with, it adds back into the Sidekiq queue, and is processed by Sidekiq just like a normal job. ! So that’s pretty much it, you have Elixir workers yet neither Sidekiq nor Rails have to know about it. ! Some people I’ve talking to have replaced bits of their Rails app with Elixir apps. !

There’s more to programming than web applications. ! There are some interesting things are being done with Elixir and Erlang.

You could fly a drone

Control a robot arm

Experiment with Parallel computing.

Best of all, you don’t have to give up Ruby.

Matz made Ruby to optimise for programmer happiness.

I have found that Elixir also makes me happy. ! I think it can make you happy too.

Thank You! www.exotpbook.com BENJAMINTANWEIHAO.GITHUB.IO [email protected] & I’m writing a book! I’m writing a book on Elixir, so if I managed to pique your interest, do check out exotpbook.com. ! Well, thank’s for listening! ! Let’s have some questions!