Speaker Deck

Modern Data Pipelines with Apache Airflow (Momentum Dev Con 2018)

by Taylor Edmiston

Published April 19, 2018 in Programming

Original presentation on Google Slides - https://docs.google.com/presentation/d/1mCgDT7DEj2jsrr09Omm4lAspihmPXaA8r-dBuyPNT5U/edit?usp=sharing

---

Abstract:

Big data needs span both business users and developers across the world. Most classical ETL and BI tools attempted to cater to this hybrid demographic resulting in cluttered GUI tools that were inflexible, inextensible, and frustrating to use. Apache Airflow takes a different approach by representing tasks and config as Python code.

Airflow is a platform to programmatically author, schedule and monitor workflows composed of arbitrary tasks run on regular schedules. Airflow provides a platform for distributed task execution across complex workflows as directed acyclic graphs (DAGs) defined by code.

Built on top of Airflow, Astronomer provides a containerized Airflow service on Kubernetes as well as a variety of Airflow components and integrations to promote code reuse, extensibility, and modularity. The core of our stack is available as cloud hosted, on prem, and is also fully open source.

---

Bio: Taylor D. Edmiston

Taylor Edmiston is a senior software engineer experienced in designing and building backend systems from web apps to APIs to platforms for startups. He has software development experience at startups from multiple top accelerators including AngelPad, Techstars, and The Brandery. Currently, he's a developer on the core team at Astronomer.io working on the customer data platform that runs batch workflows via Airflow and clickstream pipelines via Kafka on top of Kubernetes. He's in the top 25% all time on Stack Overflow having reached over 500k fellow software developers, and the top 1% on Codewars.

On a personal note, he enjoys getting stamps in his passport and has traveled to 9 countries across 4 continents so far.

---

Bio: Andy Cooper

Andy Cooper is a Software Engineer who previously focused on ETL and Business Intelligence development. More recently he has applied those skills into building a Data Engineering Platform at Astronomer.

Outside of work Andy enjoys just about any outdoor activity, including climbing, hiking, biking and skiing.