Elixir: o que pode dar errado?

@nirev
Elixir: O que pode dar
errado?
2019-05-25
Guilherme de Maio

View Slide

@nirev
@nirev

View Slide

@nirev
Baseado em São Paulo 
 
Elixir @ xerpa.com.br (2015) 
 
coorganizador SP Elixir Meetup ;) 
 
Muito Java e C antes

View Slide

@nirev
Elixir Engineer @ Telnyx 
Baseado em São Paulo 
 
Elixir @ xerpa.com.br (2015) 
 
coorganizador SP Elixir Meetup ;) 
 
Muito Java e C antes

View Slide

@nirev

View Slide

@nirev
Contexto
Coisas que normalmente associamos com a BEAM

View Slide

@nirev
Modelo de Atores

View Slide

@nirev
Processos Isolados

View Slide

@nirev
GC por Processo

View Slide

@nirev
Super escalável,
só escreve, funciona!

View Slide

@nirev
Mas.. Essa é uma visão
limitada

View Slide

@nirev
Como quebrar sua app

View Slide

@nirev
O Caso da
tonelada de átomos
1

View Slide

@nirev
Tons of atoms
Poison.decode(body, keys: :atoms!)

View Slide

@nirev
defmodule X do
def explode do
Enum.each(1 ..2_000_000, fn i ->
IO.puts(" #{i}")
20
|> :crypto.strong_rand_bytes()
|> Base.encode64()
|> String.to_atom()
end)
end
end
Tons of atoms

View Slide

@nirev
82534 ...
82535 ...
82536 ...
82537 ...
82538 ...
82539 ...
82540 ...
no more index entries in atom_tab (max=100000)
Crash dump is being written to: erl_crash.dump ...done
Tons of atoms

View Slide

@nirev
• Nomes de Modulos
• Nomes de nós
• Campos de Struct
• “decode as atom”
@nirev
Tons of atoms

View Slide

@nirev
O Caso do
Agent linkado
2

View Slide

@nirev
Linked Agent
Process 1 Linked
Process
start_link

View Slide

@nirev
Linked Agent
Process 1 Linked
Process
start_link
exit reason: exception

View Slide

@nirev
Linked Agent
Process 1 Linked
Process
start_link
exit reason: normal

View Slide

@nirev
Linked Agent
Linked
Process
“I'm a survivor”

View Slide

@nirev
Linked Agent
defmodule X do
def spawn do
spawn(fn ->
{:ok, _pid} = Agent.start_link(fn -> 42 end)
Process.sleep(1_000)
exit(:normal)
end)
end
end

View Slide

@nirev
Linked Agent
iex(1)> Process.list() |> Enum.count()
50
iex(2)> for _ <- 1 ..10, do: X.spawn
[#PID<0.94.0>, #PID<0.95.0>, #PID<0.96.0>, #PID<0.97.0>, #PID<0.98.0>,
#PID<0.99.0>, #PID<0.100.0>, #PID<0.101.0>, #PID<0.102.0>, #PID<0.103.0>]
70
iex(4)> Process.sleep(2_000)
:ok
60

View Slide

@nirev
Linked Agent
ref = Process.monitor(pid)
receive do
{:DOWN, ^ref, :process, ^pid, _reason} ->
# receive down message and do something

View Slide

@nirev

View Slide

@nirev
O Caso da
Monitoração de Requests
3

View Slide

@nirev
Request Monitoring
request
Tracker
pega dado do db
api call
atualizar status no db
publicar na ﬁla

View Slide

@nirev
Request Monitoring
request
Tracker
pega dado do db
api call
publicar na ﬁla
Tracker.add_breadcrumb(:key, metadata)
Tracker.add_breadcrumb(:update_db, %{user: x, role: y, ..})

View Slide

@nirev
Request Monitoring
request
Tracker
CRASH
breadcrumbs = Tracker.get_breadcrumbs(tracker)
send_report(exception, breadcrumbs) 
pega dado do db
api call
publicar na ﬁla

View Slide

@nirev
Request Monitoring
Pra cada request:
-Iniciar um agent
-monitorar o agent e o processo
-matar o agent quando o processo termina normalmente  
OU  
pegar os breadcrumbs e depois matar em caso de exception

View Slide

@nirev
Request Monitoring
40
80
120
160
??????

View Slide

@nirev
Request Monitoring
Cowboy implements the keep-alive mechanism by reusing the same
process for all requests. This allows Cowboy to save memory. This
works well because most code will not have any side eﬀect impacting
subsequent requests. But it also means you need to clean up if you do
have code with side eﬀects. The terminate/3 function can be used for
this purpose.

View Slide

@nirev
Request Monitoring
40
80
120
160
agents forever

View Slide

@nirev
O Caso dos
Restarts Inﬁnitos
4

View Slide

@nirev
Infinite Restarts
Process

View Slide

@nirev
Infinite Restarts
Error 
Reporter 
Task
Process

View Slide

@nirev
Infinite Restarts
Task
Supervisor
Error 
Reporter 
Task
Error 
Reporter 
Task
Process

View Slide

@nirev
Infinite Restarts
Task 
Supervisor
Error 
Reporter 
Task
Error 
Reporter 
Task
API remota caiu!!!

View Slide

@nirev
Infinite Restarts
Task 
Supervisor
supervisor(Task.Supervisor, [[name: XXX.TaskSupervisor,  
restart: :transient]])

View Slide

@nirev
Infinite Restarts
Task 
Supervisor
Error 
Reporter 
Task
Error 
Reporter 
Task

View Slide

@nirev
Infinite Restarts
Task 
Supervisor
Error 
Reporter 
Task
Error 
Reporter 
Task
API remota caiu!!!

View Slide

@nirev
Infinite Restarts
Task 
Supervisor
supervisor(Task.Supervisor, [[name: XXX.TaskSupervisor,  
restart: :temporary]])

View Slide

@nirev
O Caso do
Message Router
5

View Slide

@nirev
Message Router
Dispositivo de rastreamento

View Slide

@nirev
Backend Server
TCP Socket
A
P
I
Comandos
Message Router

View Slide

@nirev
Message
Router
Message Router
1 Genserver x Veículo

View Slide

@nirev
Message
Router
Out of Memory
Message Router

View Slide

@nirev
Message Router
Process Control Block
Stack
Heap
Process Memory 
Layout

View Slide

@nirev
Message Router
Stack
Heap 
 
Process Memory 
Layout
Shared Heap
Refc Binary
ProcBin (pointers)

View Slide

@nirev
Message Router
Stack
Heap
Private Heap Garbage Collection
Generational 
• young generation: dados recentemente alocados
• old generation: dados que sobrevivem uma 
passada de GC 
Fullsweep vs Generational: 
• min_heap_size
• fullsweep_after

View Slide

@nirev
Message Router
Shared Heap Garbage Collection
Contagem de Referências 
• todo binary sem referencias a ele será coletado
Shared Heap
Refc Binary

View Slide

@nirev
Message Router
O problema do Message Router
Recebe mensagens > 64bytes (Refc bins)
Então, uma referencia ao Large Binary ﬁca no heap
Message
Router

View Slide

@nirev
Message Router
Mas, como o router não usa muita memória
O heap não cresce mais que o min_heap_size
Message
Router

View Slide

@nirev
Message Router
Logo, referencias vivem pra sempre, e os "large
binaries" continuam no Shared Heap.
Message
Router

View Slide

@nirev
Message Router
Logo, referencias vivem pra sempre, e os "large
binaries" continuam no Shared Heap.
Message
Router
Out of Memory

View Slide

@nirev
Message Router
A SOLUÇÃO pro Message Router
1) fullsweep_after process ﬂag 
conﬁgura a frequencia do fullsweep GC, 
e isso vai coletar os ProcBins 
2) Hibernar o processo 
quando hiberna, o fullsweep GC roda 
3) Mover o trabalho para processos de vida curta,  
se possível
Message
Router

View Slide

@nirev
O que fazer quando isso
acontece

View Slide

@nirev
O que fazer quando isso
acontece
(antes de acontecer)

View Slide

@nirev
Introspecção!

View Slide

@nirev
Conectar a um nó
$ iex
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Interactive Elixir (1.8.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Node.list
[]

View Slide

@nirev
Conectar a um nó
$ iex --sname [email protected] --cookie delicia
iex([email protected])1> Node.list
[]

View Slide

@nirev
Conectar a um nó
$ iex --sname node2 --cookie delicia --remsh [email protected]
iex([email protected])1> Node.list
[:[email protected]]

View Slide

@nirev
E agora?

View Slide

@nirev
Veja o que está acontecendo!
Process.list/0
Process.info/1
:sys.get_*
MeuModulo.minha_task()

View Slide

@nirev
:observer.start()

View Slide

@nirev
https: //ferd.github.io/recon/ 
 
Recon is a library to be dropped into any other Erlang
project, to be used to assist DevOps people diagnose
problems in production nodes.

View Slide

@nirev
https: //ferd.github.io/recon/ 
 
:recon.bin_leak(3)
[
{#PID<0.80.0>, -606,
[current_function: {Process, :sleep, 1}, initial_call: {:erlang, :apply, 2}]},
{#PID<0.124.0>, -176,
[ :ssl_manager,
{:current_function, {:gen_server, :loop, 7}},
{:initial_call, {:proc_lib, :init_p, 5}}
]},
{#PID<0.1905.0>, -165,
[current_function: {:ranch_conns_sup, :loop, 4},
initial_call: {:proc_lib, :init_p, 5}
]}
]

View Slide

@nirev
Métricas!

View Slide

@nirev
VM metrics
vmstats: manda metricas pro statsd 
https: //github.com/ferd/vmstats
prometheus: manda métricas pro… prometheus
https: //github.com/deadtrickster/prometheus.ex

View Slide

@nirev

View Slide

@nirev
Visibilidade!

View Slide

@nirev
Log aggregation

View Slide

@nirev
Error Reporting
-Sentry
-Honeybadger
-Bugsnag
-Rollbar
-…

View Slide

@nirev
Moral da história?

View Slide

@nirev
Tem várias maneiras de
quebrar suas aplicações.

View Slide

@nirev
Não vá para produção  
sem visibilidade.

View Slide

@nirev
Esteja preparado!

View Slide

Guilherme de Maio 
[email protected]
Obrigado!
❤ Elixir Brasil 2019
Check us out!  
www.telnyx.com
@nirev

View Slide

Elixir: o que pode dar errado?

Elixir: o que pode dar errado?

Guilherme de Maio, nirev

More Decks by Guilherme de Maio, nirev

Other Decks in Programming

Featured

Transcript