Given at IT Weekend 2014 in Kiev
@AbeStanwayBUILDING A DATADRIVEN ORGANIZATION
View Slide
1. why2. how
“DATA IS THE NEW GOLD”
Predict the future!
Retain Customers!
Grow the business!
Recommend content!
Drive Engagement!
unclear paths to $$$
IN IT, It’s clear.
Data are Dollars
.IT Working = +$$$IT Not Working = -$$$..
How do you know ifyour IT is workingright now?
How do you know if youare earning moneyright now?
KPIs. What are they?
Etsy:Literally a Money persecond Graph
Planet Labs:Literally an Imagesper day graph
What are theKpis for kips?
$ perseconditemsbought persecond pagerequestsper seconddatabasequeries persecond memcachehits persecondfread()per second
If you do not have the dataabout your infrastructure,it is already broken.LEsson:
Test driven development->data driven development
Without data, youare flying blind
How do you know you’rehitting your goals?
How do you know if You’remaking the right ones in thefirst place?
How do you know ifyou’re still inbusiness?
How do you evenknow what planetyou live on?
Assumptions are death
You need data, yo.
1. collect2. analyze3. ???4. Profit!
1. collect2. analyze3. ACT4. Profit!
data that cannot beacted upon shouldnot be analyzed.
You are running abusiness, not anart museum
You are Trying toWin the market, nota fields medal
This can bedisappointing
Data SCientist?
Data Scientist?Realist.
Find a way to Align youremployees intellectual curiosityWith your Real business needs.LEssoN:
Train your organization
you need a data culture.
“It’s not shippeduntil it’s monitored”
“If you are not lookingat Dashboards, you arenot doing your job”
Building instrumentationand watching dashboardsare hard And Time consuming
App code -> statsD -> Graphite -> Dashboards -> Insightsby hand by hand by hand by hand by hand
Developers justwant to code
Let’sautomate
Which is easierto automate?Insights or datacollection?
Insights aresexy and fun
Collection ishard And unsexy
Collection ishard And Boring
We’re on track tohave excellentautomated insights
anomalydetection
App code -> statsD -> Graphite -> Dashboards -> Insightsby hand by hand by hand AUTOMATIC!AUTOMATIC!
(…if only we had the data)
How do weautomate datacollection?
currently haveganglia, New relic,collectD, etc
NOT WHATWE NEED
they provide data aboutyour raw machines, not yourCUSTOM DEVELOPED TECHNOLOGYAnd Application level logic
Healthy serversdon’t make you money.Healthy services do.
enterLARIMAR
Full disclosure:this is my new PROJECT!we’re going to talk about itbecause i’m pretty excited and thebeta is opening up soon.
LARIMAR uses rawmachine metrics to inferApp level architectureand inform developersabout problems
A service:cpu resourcesdisk ioPCAP dataports Usedsyscalls
A service:cpu resourcesdisk ioPCAP dataports Usedsyscallsservice fingerprintMACHINE LEARNING
A service:cpu resourcesdisk ioPCAP dataports UsedsyscallsABNORMAL BEHAVIORMACHINE LEARNING
a system:serviceserviceserviceserviceserviceMACHINE LEARNINGgraphical systemfingerprint
a system:serviceserviceserviceserviceserviceMACHINE LEARNINGAbnormal, holisticsystem behavior
Larimar automatesboth analysis AndRelevant datacollection
so your developerscan focus on codingand acting on insights
No configuration!
App code -> statsD -> Graphite -> Dashboards -> Insightsby hand AUTOMATIC!AUTOMATIC!AUTOMATIC! AUTOMATIC!
organizationalshifts are stillneeded to inspireACTION on Data
but ACTION is easier toinspire when there islots of data and lotsof insight everywhere
Create a culture whereyour developers createthese kinds of tools
When a data drivenmindset is the default,tools will buildthemselves.
Thanks!@abestanway!!larimar.io@larimarhq