Optimizing performance of Phoenix apps

My presentation at Braga BEAM User Group meetup in October 2023

Transcript

1. BERNARDO AMORIM | 2023-10-26 Optimizing performance of Phoenix Apps
2. “I made this one longer only because I did not have the leisure to make it shorter.” BLAISE PASCAL
3. Used Tools • • • • • • • • • • • • • • Environment Setup Toxiproxy - Network Latency K6 - Load Generation / Testing Ballerina Lang - Created "microservices" for testing Observability Grafana Grafana Tempo Prometheus Elixir Profiling & Benchmark fprof + erlgrind + gprof2dot + dot eflambe + speedscope Database Profiling Postgres' EXPLAIN PEV2
4. A 3-tiered application using Postgres, Ecto, LiveView (or maybe just a JSON or GraphQL API) WHAT IS A PHOENIX APP? Data Layer Application Layer Presentation Layer Postgres Phoenix Contexts + Ecto LiveView/Absinthe/Controller
5. OUR DEMO APP: MODEL posts BIGSERIAL id VARCHAR title TEXT content BIGINT topic_id TIMESTAMP inserted_at TIMESTAMP updated_at topics BIGSERIAL id VARCHAR name BIGINT parent_id TIMESTAMP inserted_at TIMESTAMP updated_at post_contributors BIGSERIAL id BIGINT post_id BIGINT user_id BOOLEAN author users BIGSERIAL id CITEXT email VARCHAR hashed_password TIMESTAMP confirmed_at TIMESTAMP inserted_at TIMESTAMP updated_at users_tokens BIGSERIAL id BIGINT user_id BYTEA token VARCHAR context VARCHAR sent_to TIMESTAMP inserted_at topic_id post_id user_id user_id parent_id
6. OUR DEMO APP
7. PERFORMANCE IMPROVEMENT LIFE CYCLE DISCOVERY PROFILE OPTIMIZE • • User Reports Monitoring • • • Distributed Tracing Lang/VM specific tools EXPLAIN ANALYZE • • • Batching / Preloading Database Indexes Faster Algorithms
8. Discovery
9. • • Unified observability standards and SDKs Combines metrics, logs, and traces OpenTelemetry • • • Trace: Lifecycle of a request across services Span: Single unit of work in a trace Event: Specific action within a span Distributed Tracing OPENTELEMETRY
10. OBSERVABILITY STACK Honeycomb.io Grafana Stack
11. • Some problems only became visible with larger data sets Data Size PRODUCTION VS DEV • Some problems only happen when there are concurrent requests coming from many users Load • Latencies between services can make some problems much worse Latency
12. Simulate Production
13. Profile N+1
14. Optimize N+1
15. Profile Slow Algorithm
16. • • Built into OTP Pair with erlgrind & gprof2dot (or qcachegrind) for nice visualization fprof • Pair with speedscope for nice flame graph visualization eflambe
17. Optimize Slow Algorithm
18. 1 Level
19. 2 Levels
20. 3 Levels
21. 4 Levels
22. 2 Wide Levels
23. Profile HOST | DATE Slow SELECT
24. Optimize HOST | DATE Slow SELECT
25. Profile HOST | DATE Slow DELETE
26. Optimize HOST | DATE Slow DELETE
27. Profile HOST | DATE GenServer Bottleneck
28. Optimize HOST | DATE GenServer Bottleneck
29. • • • • • OTEL Tracing alone will make it explicit Solution is normally batching/preloading Ecto: preload LiveView: update_many Absinthe: DataLoader N+1 SUMMARY • • • Requires lower level profiling fprof, cprof, eprof, eflambe Solution will vary, but a lot of the times some preprocessing goes a long way Slow Code • • • • Distributed tracing can help pointing out the specific query to analyse Postgres EXPLAIN will give you insight on what is wrong PEV2 will help you visualize it Solutions will vary: indexes, data denormalization, preprocessing, extra conditions to reduce result set Slow Query
30. Used Tools • • • • • • • • • • • • • • Environment Setup Toxiproxy - Network Latency K6 - Load Generation / Testing Ballerina Lang - Created "microservices" for testing Observability Grafana Grafana Tempo Prometheus Elixir Profiling & Benchmark fprof + erlgrind + gprof2dot + dot eflambe + speedscope Database Profiling Postgres' EXPLAIN PEV2
31. Thank you KEEP IN TOUCH github.com/bamorim x.com/BernardoDCGA linkedin.com/in/bdornellas