Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
InfluxDB - a distributed events and time series...
Search
Paul Dix
April 27, 2014
Technology
2k
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
InfluxDB - a distributed events and time series database
Slides from my lightning talk at the GopherCon pre-party.
Paul Dix
April 27, 2014
Transcript
71 slides
InfluxDB - a distributed time series, metrics, and events database
Paul Dix paul@influxdb.com @pauldix @influxdb
YC (W13), 3 people full time: Todd Persen John Shahid
Paul Dix (me)
What it’s for…
Metrics
Time Series
Analytics
Events
Can’t you just use a regular DB?
order by time?
Doesn’t Scale
Example from metrics: ! 100 measurements per host * 10
hosts * 8640 per day (once every 10s) * 365 days ! = 3,153,600,000 records per year
Have fun with that table…
But wait, we’ll just keep the summaries!
1h averages = ! 8,760,000 per year
Lose Detail and AdHoc Queryability
So let’s use Cassandra, HBase, or Scaleasaurus!
Too much application code and complexity
Application logic and scripts to compute summaries
Application level logic for balancing
No data locality for AdHoc queries
And then there’s more…
Web services
Libraries for web services
Data collection
Visualization
–Paul Dix “Building an application with an analytics component today
is like building a web application in 1998. You spend months building infrastructure before getting to the actual thing you want to build.”
Analytics should be about analyzing and interpreting data, not the
infrastructure to store and process it.
None
HTTP API Web services built in
HTTP API (writes) curl -X POST \ 'http://localhost:8086/db/mydb/series?u=paul&p=pass' \ -d
'[{"name":"foo", "columns":["val"], "points": [[3]]}]'
Data (with timestamp) [ { "name": "cpu", "columns": ["time", "value",
"host"], "points": [ [1395168540, 56.7, "foo.influxdb.com"], [1395168540, 43.9, "bar.influxdb.com"] ] } ]
HTTP API (queries) curl 'http://localhost:8086/db/mydb/series?u=paul&p=pass&q=.'
SQL-ish select * from events where time > now() -
1h
SQL-ish select * from “series with weird chars ()*@#0982#$” where
time > now() - 1h
Where Regex select line from application_logs where line =~ /.*ERROR.*/
and time > "2014-03-01" and time < "2014-03-03"
Only scans the time range Series and time are the
primary index
Work with many series…
Select from Regex select * from /stats\.cpu\..*/ limit 1
Downsampling on the fly…
Aggregates select percentile(90, value) from response_times group by time(10m) where
time > now() - 1d
Continuous Downsampling…
Continuous queries (summaries) select count(page_id) from events group by time(1h),
page_id into events.[page_id]
Series per page id select count from events.67 where time
> now() - 7d
Continuous queries (regex downsampling) select percentile(value, 90) as value from
/stats\.*/ group by time(5m) into percentile.90.:series_name
Percentile series per host select value from percentile.90.stats.cpu.host1 where time
> now() - 4h
Denormalization for performance
Range scans all user events for last hour select *
from events where user_id = 3 and time > now() - 1h
Continuous queries (fan out) select * from events into events.[user_id]
Series per user id select * from events.3 where time
> now() - 1h
Distributed Scale out, data locality, high availability
Raft for metadata We owe Ben Johnson a beer or
three…
Protobuf + TCP for queries, writes
Scalable Have billions of points in 1 series* or a
million different series
Libraries Go, Ruby, Javascript, Python, Node.js, Clojure, Java, Perl, Haskell,
R, Scala, CLI (ruby and node)
Visualization
Built-in UI
Grafana
Javascript library + D3, HighCharts, Rickshaw, NVD3, etc. Definitely more
to do here!
Data Collection CollectD Proxy, StatsD backend, Carbon ingestion, OpenTSDB (soon)
Coming Soon
ugh, Documentation
Series Metadata
Binary Protocol
Pubsub select * from some_series where host = “serverA” into
subscription() select percentile(90, value) from some_series group by time(1m) into subscription()
Custom Functions select myFunc(value) from some_series
Rack aware sharding and querying
Multi-datacenter replication Push and bi-directional
Indexes?
Ponies? Tell @jvshahid that you want your pony ;)
But it’s ready to go now. Production deployments already running.
Need help? support@influxdb.com Thanks! paul@influxdb.com @pauldix
More Decks by Paul Dix
See All by Paul Dix
InfluxDB IOx Project Update - 2021-02-10
pauldix
0
290
InfluxDB IOx data lifecycle and object store persistence
pauldix
1
740
InfluxDB 2.0 and Flux
pauldix
1
820
Flux and InfluxDB 2.0
pauldix
1
1.5k
Querying Prometheus with Flux
pauldix
1
1k
Flux (#fluxlang): a new (time series) data scripting language
pauldix
7
5.4k
At Scale, Everything is Hard
pauldix
2
790
IFQL and the future of InfluxData
pauldix
2
1.5k
Time series & monitoring with InfluxDB and the TICK stack
pauldix
0
560
Other Decks in Technology
See All in Technology
「個社」ではなく「業界」の課題を解く AI 対話開発 ー 価値を広げる設計と、価値を守るエンジニアリング ー
ivry_presentationmaterials
4
870
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
240
攻撃と守りは何が変わり、何が変わらないのか~最近の不正アクセスから考える、リスク評価と標準の使いどころ
oidfj
2
320
リリースしたで終わらない。 観測できるフロントエンド開発
yamakenji24
2
240
AWS re:Invent 2026 FSI Pre-Event Networking Night
junjikoide
0
210
Dark Web - Como o Cibercrime Opera na Internet
nicholsjasper
0
110
エンタープライズIAMについて - ID・権限をどう統制するか
oidfj
1
320
現場に渡し、構造で守る AI エージェントを活用したセルフサーブのデータ基盤の運用
legalontechnologies
PRO
0
110
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
140
信頼性を犠牲にせず、PR マージからデプロイまで人の介入を完全にゼロにするまでの道のり
knwoop
4
2.1k
AI-Ready なデータ基盤ってなに? & 新卒未経験が、いま考えていること
imuta
0
1.1k
さようならSSH鍵、こんにちはゼロトラスト ― AIフレンドリーで快適な自宅鯖SSH
atalie2m
14
7.5k
Featured
See All Featured
Skip the Path - Find Your Career Trail
mkilby
1
300
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
31
3.4k
Java REST API Framework Comparison - PWX 2021
mraible
34
9.8k
What Being in a Rock Band Can Teach Us About Real World SEO
427marketing
0
1.1k
Learning to Love Humans: Emotional Interface Design
aarron
275
41k
Applied NLP in the Age of Generative AI
inesmontani
PRO
4
2.6k
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
techseoconnect
PRO
0
540
Efficient Content Optimization with Google Search Console & Apps Script
katarinadahlin
PRO
1
1k
Pawsitive SEO: Lessons from My Dog (and Many Mistakes) on Thriving as a Consultant in the Age of AI
davidcarrasco
1
320
Statistics for Hackers
jakevdp
799
230k
Faster Mobile Websites
deanohume
310
32k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
11
1.1k