Light Up the Spark in Catalyst by avoiding UDFs

Processing data at scale usually results in struggling with performance, strict SLA, limited hardware etc. I've struggled with cutting Spark SQL query run-time and found the culprit! This culprit, and SOLUTION! I would like to share with you. Today in the world of Big Data and Spark we are processing high volume transactions. Catalyst is the Spark SQL query optimizer and in this talk, you will learn how to fully utilize Catalyst optimization power in order to make our queries as fast as possible, by pushing down actions and trying to avoid UDFs as much as possible and maximizing performance.

Transcript

1. Light Up the Spark in Catalyst by avoiding UDFs Adi Polak Milan | November 29 - 30, 2018
2. About me – Adi Polak @adipolak adi.polak@Microsoft.com https://medium.com/@adipolak
3. Agenda scala Apache Spark 2.3 UDF Catalyst Optimization
4. Who use Apache Spark Data scientists Data engineers Business analysts
5. Apache Spark is distributed general-purpose cluster-computing framework User-Defined Functions - Column-based functions that extend the vocabulary of Spark SQL's DSL Catalyst optimization allows some advanced programming language features that allow you to build an extensible query optimizer Catalyst - Apache Spark SQL query optimizer Concepts
6. Data analytics generic architecture
7. Structured Streaming MLib GraphFrame TensorFrames SQL SparkSession / DataFrame / Dataset APIs Data Source Connectors Catalyst Optimization & Tungsten Execution Spark Core (RDD APIs) SQL
8. Lets look at the demo
18. CATALYST
19. Fundamentals of Catalyst Optimizer SUB Attribute(x) SUB some_func(1) some_func(2) Tree Rules SUB Attribute(x) some_func(-1)
20. Spark SQL Execution Plan Logical optimization –> Optimization rules • Constant folding • Predicate pushdown • Projection pruning • … Physical Planning –> Planning strategies Catalyst Frontend Backend
21. SQL query with custom UDF
22. "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them." SQL query with custom UDF
23. "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them." SQL query with custom UDF
24. "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them." SQL query with custom UDF
25. What do we lose when using Custom UDF ? • Constant folding • Predicate pushdown
26. What can we do ?
27. Use queryExecution & explain(true) Catalyst Frontend Backend
29. Use queryExecution & explain(true) API My UDF Register
30. Lost Push Down filter
31. What can be done instead? sql functions DataFrame API: Aggregate functions Collection functions Date time functions Math functions Non-aggregate functions Sorting functions String functions Window functions sql functions Column API Expression operations..
32. How can I find what functions are available? arrayContains, minute, round, rand, spark_partition_id, isin … version
33. 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Diamonds (53,940) users_filtered (115,134) BlackFriday (302,675) UserList (537,577) Runtime Benchmarks RunTime UDF (sec) RunTime Functions (sec)
34. Takeaways • Use UDFs as a last resort • Avoid UDFs or UDAFs that perform more than one thing • Look under the hood – • Analyzing Spark’s execution plan with .explain(true)
35. Reference • https://aka.ms/AA3e7rd • https://aka.ms/AA3efgo • https://www.kaggle.com
36. GRATZIE @adipolak @adipolak Adi.polak@Microsoft.com