For analytics / ML projects Developing a data pipeline on cloud Fon Liwprasert ML Engineer at Sertis

Kamolphan Liwprasert (Fon) Machine Learning Engineer, Sertis I’m an ML engineering enthusiast with data engineering background. I love to develop solutions on cloud. 6 x GCP certified.

Every data analytics or ML project needs a good data pipeline as a foundation.

But the data is always not ready to use at the beginning.

We also need to think about automation and scalability as the project goes on.

Developing a data pipeline on cloud Specifically on Google Cloud Platform

Text & diagram slides Agenda ● Why do we need data pipeline? ● What are options on cloud? Storages / Computes / Pipelines ● Introducing Apache Airflow ● Reference Architecture + Demo code

Why do we need data pipeline? 🛠

Purposes of data pipeline Ingest data from multiple sources Transform or clean the data to ensure data quality Automation of the process

What are options on cloud? ☁

Choosing Storage Options

BigQuery Choosing Storage Options: zoom-in For analytics workload Cloud Bigtable Cloud Storage

Choosing Compute Options Google Compute Engine Google Kubernetes Engine Google Cloud Run Google App Engine Google Cloud Function IAAS : Infrastructure as a service CAAS : Container as a service PAAS : Platform as a service FAAS : Function as a service GCE GKE GAE GCF Virtual Machine Managed K8s cluster Serverless container Serverless application Serverless function platform

Choosing Data Processing Options Cloud Dataproc Processing data Workflow and scheduler Cloud Scheduler Cloud Dataflow Cloud Composer Spark or Hadoop Data processing Unified pipeline w/ Apache Beam 3 2 1 Cloud Functions or or Cloud Run Serverless options Cloud Workflows (optional)

Option 1: Low-cost & severless option Processing data (light workload) Cloud Functions Cloud Run (Or Pub/Sub) for Cloud Functions Workflow and scheduler REST API Scheduler ✓ Severless: easy & fast ✓ Low-cost solution ✓ Suitable for light workload

Option 2: Big data solution Spark or Hadoop Data processing Unified data pipeline with Apache Beam Processing data (Big data workload) Workflow and scheduler Cloud Dataproc REST API Cloud Dataflow Scheduler ✓ Big data framework: Spark, Apache Beam, Flink ✓ Scalability and reliability ✓ Opensource solutions

Option 3: Cloud Composer (Airflow) Cloud Composer Kubernetes Engine Cloud SQL Managed service + ✓ Easier for maintenance ✓ Scalability and reliability ✓ Suitable for large number of jobs that require workers

Introducing Apache Airflow

Why Apache Airflow? ● Popular open-source project for ETL or data pipeline orchestration. ● All codes are in Python. Easy to learn and use. ● Can be run locally as well for development environments.

Airflow is rich in 3rd party integration https://github.com/apache/airflo w/tree/main/airflow/providers . . . https://registry.astronomer.io/providers/

Apache Airflow UI DAGs (Job list) Calendar View Tree View

Apache Airflow basic components Sensor Wait on an event i.e. poking for a file Operator Running an action; PythonOperator Hook Interface to external services or system

Reference Architecture and Demo Code 📐📄

Reference Architecture Batch Ingestion BigQuery Cloud Storage SFTP server Cloud Composer SFTPToGCSOperator GCSToBigQueryOperator Analytics Workload BI Dashboards SFTPSensor

Text & diagram slides DAG overview SFTPSensor SFTPToGCSOperator GCSToBigQueryOperator Check if a file is available Upload that file to GCS Load file from GCS to BigQuery

Demo Time! bit.ly/airflow_gcp_demo

Text & diagram slides Simple data pipeline using Airflow (1) Import and initialize DAG bit.ly/airflow_gcp_demo ← DAG name ← Schedule ← Import necessary components

Text & diagram slides Simple data pipeline using Airflow (2) SFTPSensor for waiting for a file to be available bit.ly/airflow_gcp_demo ← SFTP Connection ← Path to file wait_for_file check-for-file SFTPSensor

Text & diagram slides Simple data pipeline using Airflow (3) SFTPToGCSOperator for uploading file(s) from SFTP to GCS bit.ly/airflow_gcp_demo ← GCP Connection ← Path to File & GCS upload_file_to_gcs upload-file-from-sftp SFTPToGCSOperator

Simple data pipeline using Airflow (4) GCSToBigQueryOperator for loading data to BigQuery from GCS source file(s) bit.ly/airflow_gcp_demo ← source GCS ← JSON schema (dictionary) ← format: CSV, AVRO, parquet load_to_bigquery load-to-bigquery GCSToBigQueryOperator

Text & diagram slides Simple data pipeline using Airflow (5) Put everything together! wait_for_file check-for-file upload_file_to_gcs upload-file-from-sftp load_to_bigquery load-to-bigquery SFTPSensor SFTPToGCSOperator GCSToBigQueryOperator bit.ly/airflow_gcp_demo ← Create DAG!

Text & diagram slides DAG graph view (from previous code)

Text & diagram slides Key Takeaways 🔑 Choosing the right solution for data pipeline depends on requirements and workload.

Text & diagram slides Thank you 😃 Let’s connect! Fon Liwprasert ML Engineer at Sertis linkedin.com/in/fonylew fonylew