Deployment of Streaming Application with Ververica Platform on Kubernetes

Kubernetes Frankfurt

Transcript

1. Rapid Deployment with Apache Flink and Ververica Platform Alexey Novakov Frankfurt, May 2023 Frankfurt Meetup
2. Alexey Novakov Product Solution Architect @ Ververica, Germany - Last 6 years working in data management - 17 years in Software Development - Distributed Systems, Big/Small/Fast Data - Astronomy, Music
3. Contents 01 Getting started with Apache Flink and VVP 02 Flink Application Lifecycle 03 VVP Integrations 04 Summary
4. 01 Getting started with Apache Flink and Ververica
5. Original Creators of Apache Flink® Enterprise Stream Processing with Ververica Platform Subsidiary of Alibaba Group About Ververica
6. Features: - High-Availability, Incremental checkpointing - Sophisticated late data handling - Low latency, High throughput - Scala, Java, SQL and Python APIs - …. and much more Apache Flink
7. Some Apache Flink Users
8. What is Ververica Platform (VVP)? Purpose-built for stateful stream processing architectures and makes operating these powerful systems easier than ever before by offering an entirely new experience for developing, deploying, and managing stream processing applications. Ververica’s mission is to ensure that developers invest their time on their core business objectives, not on maintenance and infrastructure.
9. Ververica Platform
10. VVP Components
11. Ververica Installation
12. VVP Pre-requirements Bring-your-own Kubernetes - From Cloud Providers: - AWS EKS - Azure AKS - Google GKE - On-prem cluster, OpenShift - Local development: minikube, k3s
13. VVP Helm Package $ helm repo add ververica https://charts.ververica.com $ helm -ns vvp \ install vvp ververica/ververica-platform \ --values values-vvp.yaml *see more https://docs.ververica.com/getting_started/installation.html#setting-up-the-playground
14. VVP Control Plane $ kubectl get pod -n vvp -l app=vvp-ververica-platform NAME READY STATUS RESTARTS AGE vvp-ververica-platform-75c54fcd6d-95wgh 3/3 Running 0 1m Now it is ready to run Flink applications
15. $ kubectl -ns vvp port-forward services/vvp-ververica-platform 8080:80 VVP UI: Deployments
16. 02 Flink Application Lifecycle in VVP
17. User Workflow in VVP New App Deployment Reconfigure, if needed Start Deployment Upload Flink JAR / Python Script Create Deployment Monitor Create SQL Script /
18. @main def FraudDetectionJob = val env = StreamExecutionEnvironment.getExecutionEnvironment val transactions = env .addSource(TransactionsSource.iterator) .name("transactions") val alerts = transactions .keyBy(_.accountId) .process(FraudDetector()) .name("fraud-detector") alerts .addSink(AlertSink()) .name("send-alerts") env.execute("Fraud Detection") Step 0: Build JAR file some fraud detection logic print to console case class Transaction( accountId: Long, timestamp: Long, amount: Double ) Full source code: https://github.com/novakov-alexey/flink-sandbox
19. Step 1: Upload JAR file
20. Step 2: Create New Deployment 1. Flink Session Cluster 2. Kubernetes Namespace
21. Step 2: Create New Deployment - Option 2 YAML Can be submitted via REST API also. (K8s CRD is coming soon)
22. Step 2: Create New Deployment - Option 3 REST API POST vvp-resources/deployment_target.yaml to VVP REST API to create the Deployment Target: Afterwards, you can POST vvp-resources/deployment.yaml to the REST API to create the Deployment: $ curl localhost:8080/api/v1/namespaces/default/deployment-targets \ -X POST \ -H "Content-Type: application/yaml" \ -H "Accept: application/yaml" \ --data-binary @vvp-resources/deployment_target.yaml $ curl localhost:8080/api/v1/namespaces/default/deployments \ -X POST \ -H "Content-Type: application/yaml" \ -H "Accept: application/yaml" \ --data-binary @vvp-resources/deployment.yaml
23. Step 3: Start Deployment
24. Save state for another Deployment Create new Deployment with the same configuration Monitor running Job Step 4: Manage Deployment
25. Familiar Tools for Monitoring kubectl logs
26. All Pods in VVP Namespace Optional Pods
27. 03 VVP Integration
28. VVP Universal Blob Storage • Minio is one of the option for Universal Blob Storage • Minio is useful at the development phase • Blob Storage is used for: - Code Artifacts - Flink checkpoints & savepoints https://docs.ververica.com/getting_started/installation.html#setting-up-the-playground
29. Supportted Storage Supported Storage Services AWS S3 s3:// Microsoft ABS wasbs:// Apache Hadoop® HDFS hdfs:// Google GCS gs:// Alibaba OSS oss:// Microsoft ABS Workload Identity wiaz:// values.yaml: ### Configure Minio for Universal Blob Storage vvp: blobStorage: baseUri: s3://vvp s3: endpoint: http://minio.vvp.svc:9000
30. Access Control Authenticaiton • OpenID Connect • SAML Authorization • Roles: viewer, editor, owner, admin Resource Viewer Editor Owner Admin Artifacts List, GetMetadata All All None ApiToken None None All None DeploymentDefaults Get Get All None DeploymentTarget List, Get List, Get All None Apache Flink® UI Get* All All None Namespace None None Get, Update All SecretValue None All All None All Others List, Get All All None
31. Metrics Ververica Platform bundles metrics reporters for: • Prometheus • InfluxDB • Datadog • Prometheus Pushgateway • Graphite • StatsD • SLF4J spec: template: spec: flinkConfiguration: metrics.reporters: prometheus metrics.reporter.prometheus.class: org.apache.flink.metrics.prometheus.PrometheusReporter metrics.reporter.prometheus.port: 9249
32. • Stability: Well-tested pre-configured Flink runtime • Flexibility: Deployable to your own K8s cluster • Integration: most popular 3-rd party systems are integrated (S3, AD, OAuth2, etc). • Easier Operations: practical UI and REST API. Soon K8s Operator and CRD VVP Benefits
33. To Learn More Ververica Blog: https://www.ververica.com/blog VVP documentation https://docs.ververica.com/index.html Knowledge Base: https://ververica.zendesk.com/hc/en-us
34. 34 Ververica Cloud Ververica Cloud (beta) Ultra-high performance cloud- native service for real-time data processing based on Apache Flink. Sign up for free If you do not want to run yourselves on K8s, then try …
35. Thank you info@ververica.com www.ververica.com @VervericaData