Big data for .Net Devs with Apache Spark

Slide deck of the presentation done at the .Net Dev Summit 2020 on 31 October 2020. The talk covered overview of Apache Spark, .Net for Apache Spark. The demo showed the integration between Azure Synapse Workspace which has native integration with notebooks feature to analyse the popular MovieLens datasets.

Transcript

1. Nilesh Gule @nileshgule | www.HandsOnArchitect.com Big Data for .Net Devs with Apache Spark
2. $whoami { “name” : “Nilesh Gule”, “website” : “https://www.HandsOnArchitect.com", “github” : “https://github.com/NileshGule" “twitter” : “@nileshgule”, “linkedin” : “https://www.linkedin.com/in/nileshgule”, “likes” : “Technical Evangelism, Cricket”, “co-organizer” : “Azure Singapore UG” }
4. What is Apache Spark https://spark.apache.org/
5. Apache Spark Data Sources https://posts.specterops.io/threat-hunting-with- jupyter-notebooks-part-3-querying-elasticsearch- via-apache-spark-670054cd9d47
6. Benefits of using Apache Spark • Speed • Up to 100x faster compared to Map Reduce • Ease of use • Easy to use API’s • Multi language support • 100+ operators • Unified engine • Higher level libraries & support for SQL Queries, streaming data, machine learning and graph processing • Runs everywhere • Hadoop, standalone, Mesos, Kubernetes, cloud https://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html
7. Apache Spark Components • Dataset, DataFrame, RDD • Distributed collection of data • SparkSession • Entry point into Spark API • SparkContext, SQLContext, StreamingContext unified into one • Executors • Handles distributed processing • Transformations & Actions • Transformations – lazy operations that returns immutable data structures • Actions – apply operations and return value or write data to external storage
8. Spark Common Transformations • map • flatMap • filter • Distinct • Sample(withReplacement, ..) • Union • Intersection • Subtract • cartesian • reduceByKey • groupByKey • sortByKey • Join • repartition
9. Spark Common Actions • collect • count • countByValue • Take(num) • Top(num) • Reduce(func) • Fold(zero)(func) • saveAsTextFile(path) • saveAsSequenceFile(path) • countByKey()
10. What is .Net for Apache Spark • .Net bindings for Spark written on Spark interop layer • Provides high performance bindings for C# and F# • Compliant with .Net standard https://devblogs.microsoft.com/dotnet/introducing-net-for-apache-spark/#performance
11. Demo • MovieLens Datatset • CSV files in Azure Data Lake Storage • Spark pools using Azure Synapse analytics
12. Summary • Apache Spark is great for Big Data Analytics • .Net for Apache Spark provides .Net language bindings to Spark • Azure Synapse Analytics has native support for C#
13.  Apache Spark  .Net for Apache Spark  MovieLens datasets  Azure Synapse Analytics
14. https://youtu.be/KhMKXQkIzKw https://channel9.msdn.com/Series/NET-for-Apache-Spark-101
16. Thank you very much Code with Passion and Strive for Excellence https://www.slideshare.net/nileshgule/presentations https://speakerdeck.com/nileshgule/
17. Nilesh Gule ARCHITECT | MICROSOFT MVP “Code with Passion and Strive for Excellence” nileshgule @nileshgule Nilesh Gule NileshGule www.handsonarchitect.com
18. Q&A