Big Data Envy: A Chief Data Officer's Journey

Not all agencies have exabytes of data. But size does not matter. Big Data is really about having the right data fast enough to meaningful model complex systems for cause and effect. See how one Chief Data Officer looks at the challenge.

Transcript

1. CHIEF DATA OFFICER, FCC (2010 - 2013) Creator of Sunlight Foundation’s Sunlight Labs (2006 - 2009) Greg Elin @gregelin Big Data Envy! A Chief Data Officer’s Journey
2. This is hard. No one can learn to swim for you.
3. http://gis.ncdc.noaa.gov/map/viewer/#app=cdo&cfg=cdo&theme=daily&layers=111&node=gis Some  have  it  easy  with  sensors  collec2ng  data...
4. NPRM: Notice of Proposed Rulemaking Others  must  cross  the  rainbow  bridge  of  NPRM...
5. What’s magical about Big Data anyway?
6. Big Data Envy Small Data Life Same Data Pain Data deluge Data for record keeping Analysis ready? Real time Periodic Insight for decision Too much data Too much variation Too costly to move Loosely structured Loosely structured Costly to standardize Heterogeneity Heterogeneity Scaling challenges Distributed across networks Distributed across networks Permission nightmare Complex interrelationships Complex interrelationships Constantly redefined Diverse sources Diverse sources Time to organize “Big Data = Difficult to capture, store, search, share and analyze” NIST
7. Defining Big Data Data sufficient in scope, detail and timeliness to model dynamic system for cause and effect. See. Predict. Choose. Influence.
8. (~7,000 panelists across 13 ISPs early-2011)
9. Observation = Situational Awareness 1 2 3
10. Boyd’s OODA Loop Deming Cycle PDCA Motorcycle Safety SIP-DE Observe Orient Decide Act ! Plan Do Check Act (Adjust) Scan Identify Predict Decide Execute Big Data supports situational awareness decision making. See. Predict. Choose. Influence.
11. OODA: make two moves for each move your opponent makes.
12. source G ene Kim
13. Silo’ed  IT  too  slow.  Enter  DevOps. Adam Jacobs
14. @FCC
15. Modernize and streamline how the FCC: Collects Data EFFICIENT, TIMELY, RELEVANT Uses and Manages Data STANDARDIZE AND MODERNIZE Shares Data TRANSPARENT, USEFUL PUBLIC DATA TOOLS
16. Individuals responsible for creating policies, practices and tools to make data an asset for daily use
17. Centralized data set inventory for smarter tracking and licensing Cloud-services to reduce cost of development and maintenance
18. photo: Creative Commons / Dan MS API’s bridge data islands providing continuous 2 way traffic
19. Answer  a  Ques2on  a  with  a  URL (via  API)
20. http://ach.fcc.gov
21. github.com/fcc Answer  a  Ques2on  a  with  a  URL
22. Roadmap  to  Stats  (APIs  /  Dashboards) 1.Show  examples  and  discuss   2.Iden2fy  just  6  to  12  ques2ons   3.Gather  the  SQL  statements  into  a  doc   4.Pick  friendly  URL  to  display  answers   5.Repeat  and  extend
23. Adam Jacobs Infrastructure  as  Code  /  Data  as  Code I  <heart>  DevOps