Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Open Source Data Tooling Landscape
Search
Carol Willing
PRO
August 24, 2021
Technology
1
64
The Open Source Data Tooling Landscape
Given for Coiled webinar on August 24, 2021.
Carol Willing
PRO
August 24, 2021
Tweet
Share
More Decks by Carol Willing
See All by Carol Willing
Thriving with Python: Navigate the pitfalls in a polyglot world
willingc
PRO
0
33
Pragmatic Python: Python 3.12 and beyond
willingc
PRO
0
11
The Future is Notebooks
willingc
PRO
0
68
PyCon 2023 Keynote
willingc
PRO
0
160
Python: The People's Programming Language
willingc
PRO
0
100
A Random Walk with Snakes and Friends
willingc
PRO
0
45
Jupyter Notebooks for Humans
willingc
PRO
0
220
Learning Python
willingc
PRO
0
96
Play with Code: The Joy of p5.js
willingc
PRO
0
300
Other Decks in Technology
See All in Technology
プラットフォームってつくることより計測することが重要なんじゃないかという話 / Platform Engineering Meetup #8
taishin
1
360
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
長期間TiDBを使ってきた話 @ 私たちはなぜNewSQLを使うのかTiDB選定5社が語る選定理由と活用LT / Experiences with TiDB Over Time
chibiegg
2
900
障害対応をちょっとずつよくしていくための 演習の作りかた
heleeen
0
200
Databricks における 『MLOps』
databricksjapan
2
170
ServiceNow Knowledge Learning Rise up
manarobot
0
210
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
170
MapLibreとAmazon Location Service
dayjournal
1
150
オーナーシップを持つ領域を明確にする
konifar
13
3.2k
非同期推論システムによるコスト削減と信頼性向上
koki_nishihara
0
250
Azure犬駆動開発の記録/GlobalAzureFukuoka2024_20240420
nina01
1
210
LangSmith入門―トレース/評価/プロンプト管理などを担うLLMアプリ開発プラットフォーム
os1ma
3
140
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Fontdeck: Realign not Redesign
paulrobertlloyd
76
4.9k
10 Git Anti Patterns You Should be Aware of
lemiorhan
648
58k
Web development in the modern age
philhawksworth
202
10k
Clear Off the Table
cherdarchuk
84
310k
Infographics Made Easy
chrislema
238
18k
Embracing the Ebb and Flow
colly
80
4.1k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Product Roadmaps are Hard
iamctodd
44
9.7k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
17
1.4k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
78
42k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
60
14k
Transcript
The Open Source Data Tooling Landscape Carol Willing VP of
Learning Noteable web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Headline Slide Sub-headline The 10 Best Practices for Remote Software
Engineering Focusing on the human element of remote software engineer productivity Vanessa Sochat DOI:10.1145/3459613 Attribution: xkcd 1 Today
Common Data Challenges Exploring Solutions with Open Source Data Tools
2 Data
SCALE
SPEED
CONNECTIONS
CHOICES
The Data Pipeline Perspectives Attribution: Red Bull 3 People
The Data Pipeline Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Executives Opportunity and
Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear Users Productivity and Needs
Attribution: Red Bull Start small...
@WillingCarol 14 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l ...and scale.
Open Source Data Tooling Landscape 4 Ecosystem
Python R Julia Fortran SQL C++ Go Rust Java Scala
4 Ecosystem Programming Languages JavaScript TypeScript Data Analysis Workflows Interactivity
4 Ecosystem Data Work fl ow Project Definition Data Collection
Computation and Modeling Evaluation Deploy at Scale Monitoring Data Preparation Exploratory Analysis Share Results Revisit Goals
Challenges ‣ Foundation (existing infrastructure to cloud) ‣ Variability (DIY
to Hosted/Managed Service) ‣ Complexity ‣ Language ecosystems ‣ Growth
Challenges (cont.) ‣ Best practices / de facto standards ‣
Jargon ‣ Abstractions ‣ Hype CRISP-DM Attribution: IBM Cross-industry standard process for data mining 1996
4 Ecosystem Taxonomy Business Goals People Ethics Model creation Training
Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical
4 Ecosystem Julia Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical DrWatson.jl ParameterSchedulers.jl Pluto.jl IJulia JupyterLab nteract VSCode Plots.jl (Viz) Gadfly.jl (Viz) Makie.jl (Viz - GPU) Flux.jl (ML) Knet.jl (ML/BL) MLJ.jl (ML) Mocha.jl (ML/DL) Tensorflow.jl (ML/DL wrapper) JuMP (optimization) Dataframes.jl ProgressMeters.jl
4 Ecosystem Python Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical Dask JupyterHub Binder Kubernetes papermill Dagster Airflow prefect scipy statsmodel JupyterLab nteract VSCode matplotlib seaborn altair plotly numpy scikit-learn pytorch tensorflow pandas PyJanitor dask datasette evidently bokeh panel voila dash python scripts napari geopandas feast keras fastai fairlearn
4 Ecosystem R Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical RStudio JupyterLab IRkernel ggplot tidyverse dplyr tidyr lubridate readr readxl googlesheets4 ggplot2 rmarkdown Shiny plumber purrr reticulate Keras Tensorflow sparklyr ropensci.org knitr forcats mlr3 CNTK theanos
Algorithmic Business Thinking (ABT) 5 Management Paul McDonagh-Smith MIT Sloan
School of Management https://mitsloan.mit.edu/faculty/directory/paul-mcdonagh-smith https://www.youtube.com/watch?v=bqtn2tYg-kw
@WillingCarol 25 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l Got data at scale? Use open source tools.
web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Thank you The Open Source Data Tooling Landscape Carol Willing VP of Learning Noteable
6 Additional Resources https://krzjoa.github.io/awesome-python-data-science/#/ https://github.com/FavioVazquez/ds-cheatsheets https://www.the-modeling-agency.com/crisp-dm.pdf https://github.com/academic/awesome-datascience