Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Discover the Latest Innovations in Apache Airfl...

Discover the Latest Innovations in Apache Airflow 3.0, Designed to Enhance Data Orchestration for Teams of All Sizes

目的: Airflow 3.0の主要なアップデートを紹介し、現ユーザーと新規ユーザーの両方にとっての利点を強調する。
内容:
- Airflowの紹介とデータパイプラインにおけるその役割。
- Airflow 3.0の主な機能: モダンなUI、DAGのバージョン管理、タスクの分離、複数言語のサポート。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1.  8IBU`T"JSGMPX  "JSGMPX"SDIJUFDUVSF  'MFYJCMF5JNFUBCMF  6*.PEFSOJ[BUJPO  %"(7FSTJPOJOH

     *NQSPWFE#BDLGJMM  %BUB"TTFUT"TTFU"XBSF4DIFEVMJOH  &WFOUESJWFOTDIFEVMJOH  4DFOBSJP  $PODMVTJPO "HFOEB
  2. "JSGMPX%"( You define each task as a Python function or

    an operator and then organize these tasks into a Directed Acyclic Graph (DAG) to manage dependencies and execution order. You can set a scheduled time similar to a cron job.
  3. "JSGMPX5BTL-PH 5BTL`TEFUBJMFEMPHT If you want to see detailed logs of

    a task, simply click on the log to view it. Airflow will then open the full log view, so you can quickly understand what happened.
  4. "JSGMPX"SDIJUFDUVSF Airflow 2 Airflow 3 "JSGMPXWT"JSGMPX read The system turns

    the code into metadata write data create a new DagRun push job run the actual code limit scalability and security Overall, Airflow 3 puts the API in the center, separates components, reduces database load, and supports remote or cloud-native deployment. update task status cache frequent queries easing database load boosting overall throughput reduce the chance of misuse or attacks These nodes can live outside the cluster.
  5. %FWFMPQFS1SPEVDUJWJUZ • 'MFYJCMF 5JNFUBCMFT • %"(7FSTJPOJOH • *NQSPWFE#BDLGJMM $PTU&GGJDJFODZ •

    &WFOUESJWFO TDIFEVMJOH • 6*JNQSPWFNFOUT BOEQBSBMMFM%"( SVOT #FUUFS %BUB (PWFSOBODF • "TTFUTBOEBTTFU ESJWFOTDIFEVMJOH • "TTFU8BUDIFS  "TTFU&WFOU $PODMVTJPO