Slide 1

Slide 1 text

Anirudh Ramananathan ([email protected]) Software Engineer (Kubernetes) Timothy Chen ([email protected]) Co-founder & CTO (HyperPilot) Apache Spark on Kubernetes

Slide 2

Slide 2 text

Agenda • Kubernetes & Containers • Motivation • Design • Demo • Deep Dive • Roadmap

Slide 3

Slide 3 text

What is Kubernetes?

Slide 4

Slide 4 text

Kubernetes Kubernetes is an open-source system

Slide 5

Slide 5 text

Kubernetes Kubernetes is an open-source system for automating deployment, scaling, and management

Slide 6

Slide 6 text

Kubernetes Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

Slide 7

Slide 7 text

‘Containerized’

Slide 8

Slide 8 text

Containers libs app kernel libs app libs app libs app • Repeatable Builds and Workflows • Application Portability • High Degree of Control over Software • Faster Development Cycle • Reduced dev-ops load • Improved Infrastructure Utilization

Slide 9

Slide 9 text

• Large OSS Community - 1200+ contributors and 45k+ commits • Ecosystem and Partners - 100+ organizations involved • One of the top 100 projects overall on GitHub - 23k+ stars Kubernetes

Slide 10

Slide 10 text

Overview

Slide 11

Slide 11 text

At a Glance kubelet UI kubelet CLI API users master nodes etcd kubelet scheduler controllers apiserver

Slide 12

Slide 12 text

Nodes and Pods Pod Volume Containers Pod Containers 80 80 8080 8080 Volume Node • A pod is a set of co-located containers • Created by a declarative specification supplied to the master • Each pod has its own IP address • Volumes can be local or network-attached

Slide 13

Slide 13 text

Motivation

Slide 14

Slide 14 text

Why Spark on Kubernetes? • Resource sharing between batch, serving and stateful workloads – Streamlined developer experience – Reduced operational costs – Improved infrastructure utilization • Kubernetes and the Container Ecosystem – Lots of addon services: third-party logging, monitoring, and security tools – For example, the Istio project, announced May 24, by IBM, Google and Lyft, provides a service mesh for authenticating, authorizing, tracing, and timing, and rate-limiting container-to-container communication, and more.

Slide 15

Slide 15 text

Design

Slide 16

Slide 16 text

Spark, meet Kubernetes! Spark Core Kubernetes Standalone YARN Mesos GraphX SparkSQL MLlib Streaming

Slide 17

Slide 17 text

Spark, meet Kubernetes! Spark Core Kubernetes Scheduler Backend Kubernetes Cluster new executors remove executors configuration • Resource Requests • Authnz • Communication with K8s

Slide 18

Slide 18 text

Kubernetes, meet Spark! Kubernetes Cluster File Staging Server • Staging server: component to stage local files • Spark Shuffle service: component to store shuffle data for dynamic allocation • ThirdParty/CustomResources: extend Kubernetes API with Spark Knowledge Shuffle Service SparkJob API Endpoint

Slide 19

Slide 19 text

Kubernetes Integration Dependencies Container images with dependencies baked in Files from GCS/S3/HDFS/HTTP File Staging Server Staged files and JARs Several ways of running Spark Jobs along with their dependencies on Kubernetes

Slide 20

Slide 20 text

Administration Namespaces Resource Accounting Logging Monitoring Resource Quota Pluggable Authorization Admission Control RBAC • Launch Spark Jobs as a particular user into a specific namespace • RBAC and Namespace-level resource quotas • Audit logging for clusters • Several monitoring solutions to see node, cluster and pod-level statistics

Slide 21

Slide 21 text

Focus Areas Wordcloud of the command-line options we added to spark-submit on Kubernetes

Slide 22

Slide 22 text

Demo

Slide 23

Slide 23 text

Deep Dive

Slide 24

Slide 24 text

Deep Dive spark-subm it kubernetes cluster apiserver scheduler • Spark Submit submits job to K8s

Slide 25

Slide 25 text

• Spark Submit submits job to K8s • K8s schedules the driver for job Deep Dive kubernetes cluster apiserver scheduler schedule driver pod spark driver

Slide 26

Slide 26 text

• Spark Submit submits job to K8s • K8s schedules the driver for job Deep Dive • Spark Submit submits job to K8s • K8s schedules the driver for job • Driver requests executors as needed kubernetes cluster apiserver scheduler spark driver create executor pods

Slide 27

Slide 27 text

• Spark Submit submits job to K8s • K8s schedules the driver for job Deep Dive • Spark Submit submits job to K8s • K8s schedules the driver for job • Driver requests executors as needed • Executors scheduled and created kubernetes cluster apiserver scheduler spark driver schedule executor pods executors

Slide 28

Slide 28 text

• Spark Submit submits job to K8s • K8s schedules the driver for job Deep Dive • Spark Submit submits job to K8s • K8s schedules the driver for job • Driver requests executors as needed • Executors scheduled and created • Executors run tasks kubernetes cluster apiserver scheduler spark driver executors

Slide 29

Slide 29 text

• Spark Submit submits job to K8s • K8s schedules the driver for job Deep Dive • Spark Submit submits job to K8s • K8s schedules the driver for job • Driver requests executors as needed • Executors scheduled and created • Executors run tasks • Driver “completes” job and persists logs kubernetes cluster apiserver scheduler spark driver

Slide 30

Slide 30 text

Roadmap

Slide 31

Slide 31 text

Spark Streaming Spark Roadmap Spark Shell Client Mode Python/R support Cluster Mode Java/Scala Support Dynamic Allocation Local File Staging High Availability Spark SQL GraphX MLlib Dec 2016 Development Began Mar 2017 Alpha Release June 2017 Beta Release Nov 2016 Design = supported but untested = not yet supported

Slide 32

Slide 32 text

We’re just getting started... • Kubernetes CustomResources • Priorities and Preemption for Pods • Batch Scheduling and Resource Sharing • Cluster Federation and Multi-cloud deployments • Ecosystem: Kafka, Cassandra, HDFS, etc

Slide 33

Slide 33 text

Contributors Organizations Alphabetically: • Google • Haiwen • Hyperpilot • Intel • Palantir • Pepperdata • Red Hat Links: • Spark 2.2.0 Documentation • https://issues.apache.org/jira/browse /SPARK-18278 • https://github.com/kubernetes/kubern etes/issues/34377

Slide 34

Slide 34 text

Try it out today! https://github.com/apache-spark-on-k8s/spark

Slide 35

Slide 35 text

Thank You. HDFS on Kubernetes - Lessons Learned June 7 at 11:00 AM in Room 2003 Join us Wednesdays at 10am PT at the SIG BigData meeting https://github.com/kubernetes/community/