Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Big data and Machine learning APIs
Search
Sam Bessalah
December 03, 2014
Technology
320
4
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Big data and Machine learning APIs
Sam Bessalah
December 03, 2014
Transcript
58 slides
Big Data and Machine Learning APIs
Sam Bessalah @samklr Software Engineer, Freelance Data Engineering, Distributed systems,
Machine Learning Paris Data Geek Meetup @DataParis me :
None
None
None
Big Data Legends ….
Big Data Legends … Web logs Sensors Other Data source
.. . . .
A Big Data Legend … Web logs Sensors Other Data
sources .. . . .
A Big Data Legend … Web logs Sensors Other Data
sources .. . . .
A Big Data Legend … Web logs Sensors Other Data
sources .. . . .
A Big Data Legend … Web logs Sensors Other Data
sources .. . . .
A Big Data Legend … Web logs Sensors Other Data
sources .. . . . Data Driven Decisions Smart Applications
BUT ….
- Building big data infrastructures is no easy task. -
Leveraging data for decision making requires a mix of multiples skills : . System Engineering . Distributed computing . Statistics . Machine Learning
Solutions …. - Build Data platforms as a service. -
Build robust and consistent APIs to bring big data to the masses. - Leverages fluent APIs for fast data science
None
Big Data is not just about throwing data to Hadoop.
It’s also about data pipelines
Data Sources
Data Sources
Data Sources - High Throughput distributed mssaging platform - Publish
Subscribe Model - Modelled as a distributed replicated log - Persists messages to disk - Categorizes messages into Topics - Allows message retention for long specified amount of time - Allows stream replay in case of failure
Data Sources Machine Learning High Latency Batch Apps Real Time
Processing
How do you build an API around that?
None
/ingest REST API
/ingest
/ingest /query /trainModel /process
Things to be careful with - Multitenancy (Yarn, Mesos, Docker…)
- Job Scheduling - Security - Serialisation : ProtoBuf, Thrift, Avro - Storage Format : Optimize queries with columnar storage. - Compression : LZO, Snappy
Making sense of data …
None
What is Machine Learning?
http://dilbert.com/strips/comic/2013-02-02
None
https://speakerdeck.com/nivdul/lightning-fast-machine-learning-with-spark-1
Machine Learning workflow
Machine Learning workflow Text, Images, etc
Machine Learning workflow Text, Images, etc Feature Extraction
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Training
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Training Predictive Model
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Training Predictive Model New Data Feature Vector Prediction
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Predictive Model New Data Prediction
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Predictive Model New Data Prediction BLACK BOX
Machine Learning Libraries and Frameworks
scikit-learn.org
Text, Images, etc Feature Extraction Predictive Model New Data Prediction
X = vect.fit_transform(input) clf.fit(X,y) X_new = vect.fit_transform(input) y_new= clf.predict(X_new)
http://arxiv.org/abs/1309.0238
From library to web APIs
Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm
Predictive Model New Data Prediction BLACK BOX
Machine Learning workflow Text, Images, etc Transformed Data Application Prediction
Predictive API
Predictive Web APIs
Some examples
Challenges of Predictive APIs
http://www.r-bloggers.com/data-science-toolbox-survey-results-surprise-r-and-python-win/
Modeling and Prediction are just a small part of the
process
- Data locality and data gravity - Support the full
workflow - Verticalization of platforms - Scalability - Collaboration and interoperability - Black boxing of implementations
Explore machine learning for APIs orchestration. Talk to Ori @OriPekelman
Next Frontier ? Or actual reality ?
None
http://speakerdeck.com/samklr
More Decks by Sam Bessalah
See All by Sam Bessalah
Streaming Platforms
samklr
0
400
Intro to Parquet (June 2015)
samklr
0
360
High Performance RPC with Finagle
samklr
1
250
Dotscale 2015 Lightning - Distributed Systems Research
samklr
1
870
Datageeks_27-05.pdf
samklr
0
95
Scalable Machine Learning
samklr
2
290
mesos.devoxx.2014
samklr
2
320
Algebird : Abstract Algebra for Big Data Analytics.
samklr
9
3k
Algebra for analytics
samklr
1
330
Other Decks in Technology
See All in Technology
2026-11-16 JAWS-UG コンテナ支部 #31 ゆるLT大会 ECR のライフサイクルポリシーを いい感じに設定したい / ECR Lifecycle policy
masasuzu
3
620
その修正で本当に速くなったのか? k6とOpenTelemetryで確かめる性能改善 / Performance review with k6 and OpenTelemetry
ymotongpoo
3
440
常時稼働エージェントの作り方(Claude公式ブログを資料化)
minorun365
PRO
19
29k
ポケモンバトルのAIエージェント
koji_kawamura
0
220
From vibe coding to reliable (agentic) development
jcabot
0
240
フロントエンドのテストカバレッジは 100%強制がいい
bengo4com
3
1.3k
New Relicのアラート対応をClaude Codeで自動修正するワークフローを作った話
manabuseki
2
620
「個社」ではなく「業界」の課題を解く AI 対話開発 ー 価値を広げる設計と、価値を守るエンジニアリング ー
ivry_presentationmaterials
4
860
Kansai WordPress Meetup@大阪『ブロックテーマを作る上で知っておくべきこと
torounit
0
310
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
700
プラガブルで宣言的なソフトウェアファクトリーを支えるTypeScript
timakin
1
360
NutanixへのSSHアクセスの今後について
shadowhat
0
170
Featured
See All Featured
Unsuck your backbone
ammeep
672
58k
Impact Scores and Hybrid Strategies: The future of link building
tamaranovitovic
0
490
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
greggifford
PRO
0
410
Money Talks: Using Revenue to Get Sh*t Done
nikkihalliwell
0
570
Lightning Talk: Beautiful Slides for Beginners
inesmontani
PRO
2
860
How Fast Is Fast Enough? [PerfNow 2025]
tammyeverts
3
990
BBQ
matthewcrist
90
10k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
livdayseo
1
250
Ten Tips & Tricks for a 🌱 transition
stuffmc
1
290
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
23k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
287
14k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
35
2.6k