Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DataKorea-Python_Data_생태계_오늘과_내일

 DataKorea-Python_Data_생태계_오늘과_내일

Sungmin Han

October 15, 2023
Tweet

More Decks by Sungmin Han

Other Decks in Technology

Transcript

  1. Python Data 생태계
    오늘과 내일
    Data Korea

    View full-size slide

  2. Data architecture trends
    Data Processing Logging Analysis Visualize AI

    View full-size slide

  3. Google Trend: Python Data Framework
    2016 3Q - 2021 3Q (5yr)

    View full-size slide

  4. GitHub Stars: Python Data Framework
    2010 ~ 2021
    For Open source Project

    View full-size slide

  5. Incubator & Foundation
    Apache Foundation
    Apache Airflow
    Apache Superset
    Apache Hadoop
    Apache Hive
    Apache Tez
    Apache Kafka
    Apache Drill
    Apache Beam
    Apache Cassandra
    Apache Impala
    Apache Hudi
    Apache Flink Apache Spark Apache Parquet
    Apache Hue Apache CouchDB And many other things...

    View full-size slide

  6. Data Eco-system in Enterprize
    Databricks
    Apache Spark Delta Lake Redash
    Koalas

    View full-size slide

  7. Data Eco-system in Enterprize
    Spark
    Spark SQL Spark Streaming MLlib GraphX
    SparkR PySpark

    View full-size slide

  8. Data Eco-system in Enterprize
    Elastic
    Elasticsearch Kibana Logstash Beats
    Observability Security Elastic Cloud

    View full-size slide

  9. Data Eco-system in Enterprize
    BigQuery Data Flow Vertex AI Firebase Realtime Database
    TFDV
    Google
    Firebase Firestore Dataproc Dataprep
    Pubsub Bigtable Spanner Data Studio
    Google Dremel F1 Data Fusion
    Datalab Dataset Search And many other things...

    View full-size slide

  10. Data Eco-system in Enterprize
    Presto
    Facebook
    Prophet RocksDB ZippyDB
    LogDevice OpenBIC fbzmq NeuralCompression
    Haxl

    View full-size slide

  11. Data Eco-system in Enterprize
    LinkedIn
    Star-tree Apache Pinot Apache Kafka
    Apache Samza Apache DataFu Cubert
    Dr. Elephant Voldemort White Elephant
    Gobblin

    View full-size slide

  12. Data Eco-system in Enterprize
    Uber
    Petastorm Apache Hudi vis.gl
    Marmaray
    deck.gl AVS M3 H3 AresDB
    kepler.gl

    View full-size slide

  13. Data Eco-system in Enterprize
    Airbnb
    Airpal Apache Airflow visx
    Apache Superset
    Omniduct Aerosolve ReAir

    View full-size slide

  14. Data Eco-system in Enterprize
    Spotify
    luigi scio heroic annoy
    ratatool pyschema styx
    hdfs2cass featran big-data-rosetta-code

    View full-size slide

  15. The world of Database
    RDBMS NoSQL GraphDB Specialized DB
    MySQL
    PostgreSQL
    MongoD
    B
    Dgraph InfluxDB
    MariaDB
    Oracle
    MSSQL
    Cassandra
    Redis
    hbase
    Couchbase
    DynamoDB
    Cayley
    Arangodb
    Neo4j
    Orientdb
    Janusgraph
    RethinkDB
    AresDB
    Spanner

    View full-size slide

  16. Data Eco-system in sight
    Enterprize Open Source Cloud

    View full-size slide

  17. Data with AI
    Feature Store Data Validation Dataset Catalog
    Tecton
    Feast TFDV DVC MetaCat
    Hopswork
    Vertex AI
    michelangelo
    alibi-detect
    scikit-multiflow
    TorchDrift
    Hub Apache Atlas
    DataHub
    amundsen
    megda

    View full-size slide

  18. Introduce Data Korea Community
    https://www.facebook.com/groups/362069702038744
    Facebook Group
    since 2021.07.23

    View full-size slide