Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
InfluxDB - a distributed events and time series database
Search
Paul Dix
April 27, 2014
Technology
1
1.9k
InfluxDB - a distributed events and time series database
Slides from my lightning talk at the GopherCon pre-party.
Paul Dix
April 27, 2014
Tweet
Share
More Decks by Paul Dix
See All by Paul Dix
InfluxDB IOx Project Update - 2021-02-10
pauldix
0
180
InfluxDB IOx data lifecycle and object store persistence
pauldix
1
480
InfluxDB 2.0 and Flux
pauldix
1
590
Flux and InfluxDB 2.0
pauldix
1
1.2k
Querying Prometheus with Flux
pauldix
1
680
Flux (#fluxlang): a new (time series) data scripting language
pauldix
7
4.8k
At Scale, Everything is Hard
pauldix
2
630
IFQL and the future of InfluxData
pauldix
2
1.2k
Time series & monitoring with InfluxDB and the TICK stack
pauldix
0
350
Other Decks in Technology
See All in Technology
Python と Snowflake はズッ友だょ!~ Snowflake の Python 関連機能をふりかえる ~
__allllllllez__
1
120
リテール金融(キャッシュレス・ネット銀行・ネット証券)の競争環境と経済圏
8maki
0
1.2k
JSON攻略法.pdf
miyakemito
8
5.1k
Google Cloud Next '24 Recap(Cloud Run/k8s)
mokocm
0
240
DevOpsメトリクスとアウトカムの接続にトライ!開発プロセスを通して計測できるメトリクスの活用方法
ham0215
2
240
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
260
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
5
530
FrontDoorとWebAppsを組み合わせた際のリダイレクト処理の注意点
kenichirokimura
1
530
SIEMを用いて、セキュリティログ分析の可視化と分析を実現し、PDCAサイクルを回してみた
coconala_engineer
0
340
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
130
Terraformあれやこれ/terraform-this-and-that
emiki
8
1.4k
Featured
See All Featured
Raft: Consensus for Rubyists
vanstee
132
6.3k
A Tale of Four Properties
chriscoyier
151
22k
A designer walks into a library…
pauljervisheath
200
23k
Automating Front-end Workflow
addyosmani
1356
200k
Creatively Recalculating Your Daily Design Routine
revolveconf
210
11k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
60
14k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Building a Modern Day E-commerce SEO Strategy
aleyda
17
6.4k
Robots, Beer and Maslow
schacon
PRO
155
7.9k
Optimizing for Happiness
mojombo
370
69k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
2
1.3k
Building Applications with DynamoDB
mza
88
5.6k
Transcript
InfluxDB - a distributed time series, metrics, and events database
Paul Dix paul@influxdb.com @pauldix @influxdb
YC (W13), 3 people full time: Todd Persen John Shahid
Paul Dix (me)
What it’s for…
Metrics
Time Series
Analytics
Events
Can’t you just use a regular DB?
order by time?
Doesn’t Scale
Example from metrics: ! 100 measurements per host * 10
hosts * 8640 per day (once every 10s) * 365 days ! = 3,153,600,000 records per year
Have fun with that table…
But wait, we’ll just keep the summaries!
1h averages = ! 8,760,000 per year
Lose Detail and AdHoc Queryability
So let’s use Cassandra, HBase, or Scaleasaurus!
Too much application code and complexity
Application logic and scripts to compute summaries
Application level logic for balancing
No data locality for AdHoc queries
And then there’s more…
Web services
Libraries for web services
Data collection
Visualization
–Paul Dix “Building an application with an analytics component today
is like building a web application in 1998. You spend months building infrastructure before getting to the actual thing you want to build.”
Analytics should be about analyzing and interpreting data, not the
infrastructure to store and process it.
None
HTTP API Web services built in
HTTP API (writes) curl -X POST \ 'http://localhost:8086/db/mydb/series?u=paul&p=pass' \ -d
'[{"name":"foo", "columns":["val"], "points": [[3]]}]'
Data (with timestamp) [ { "name": "cpu", "columns": ["time", "value",
"host"], "points": [ [1395168540, 56.7, "foo.influxdb.com"], [1395168540, 43.9, "bar.influxdb.com"] ] } ]
HTTP API (queries) curl 'http://localhost:8086/db/mydb/series?u=paul&p=pass&q=.'
SQL-ish select * from events where time > now() -
1h
SQL-ish select * from “series with weird chars ()*@#0982#$” where
time > now() - 1h
Where Regex select line from application_logs where line =~ /.*ERROR.*/
and time > "2014-03-01" and time < "2014-03-03"
Only scans the time range Series and time are the
primary index
Work with many series…
Select from Regex select * from /stats\.cpu\..*/ limit 1
Downsampling on the fly…
Aggregates select percentile(90, value) from response_times group by time(10m) where
time > now() - 1d
Continuous Downsampling…
Continuous queries (summaries) select count(page_id) from events group by time(1h),
page_id into events.[page_id]
Series per page id select count from events.67 where time
> now() - 7d
Continuous queries (regex downsampling) select percentile(value, 90) as value from
/stats\.*/ group by time(5m) into percentile.90.:series_name
Percentile series per host select value from percentile.90.stats.cpu.host1 where time
> now() - 4h
Denormalization for performance
Range scans all user events for last hour select *
from events where user_id = 3 and time > now() - 1h
Continuous queries (fan out) select * from events into events.[user_id]
Series per user id select * from events.3 where time
> now() - 1h
Distributed Scale out, data locality, high availability
Raft for metadata We owe Ben Johnson a beer or
three…
Protobuf + TCP for queries, writes
Scalable Have billions of points in 1 series* or a
million different series
Libraries Go, Ruby, Javascript, Python, Node.js, Clojure, Java, Perl, Haskell,
R, Scala, CLI (ruby and node)
Visualization
Built-in UI
Grafana
Javascript library + D3, HighCharts, Rickshaw, NVD3, etc. Definitely more
to do here!
Data Collection CollectD Proxy, StatsD backend, Carbon ingestion, OpenTSDB (soon)
Coming Soon
ugh, Documentation
Series Metadata
Binary Protocol
Pubsub select * from some_series where host = “serverA” into
subscription() select percentile(90, value) from some_series group by time(1m) into subscription()
Custom Functions select myFunc(value) from some_series
Rack aware sharding and querying
Multi-datacenter replication Push and bi-directional
Indexes?
Ponies? Tell @jvshahid that you want your pony ;)
But it’s ready to go now. Production deployments already running.
Need help? support@influxdb.com Thanks! paul@influxdb.com @pauldix