How existing Big Data technologies are impacting Next Generation ISR

Subtitle: Intelligence, Surveillance, and Reconnaisance (ISR) analysts are no longer alone in the quantity and variability of data they need to process to make decisions

This talk was given at the "Next Generation ISR" Summit on Thursday, December 11.

Background / Landscape:

* Other industries’ data collection and processing has now matched ISR scale.
* Other commercial sectors have similar needs for intelligence integration (e.g. marketing intel, sales intel, consumer intel, etc.)
* The government & military are paying for cross-industry big data solutions.

Key Takeaways:
* Data collection is mostly a solved problem and getting cheaper and easier
* Data harmonization, homogenization, transport, ingestion, and integration are key areas of concern
* Next generation big data tools in open source and commercial sectors can now be directly applied to next generation ISR needs with less customization than ever before
* Government and military buyers have the procurement muscle to force vendors to adapt

Transcript

1. How existing Big Data technologies are impacting Next Generation ISR Intelligence, Surveillance, and Reconnaisance (ISR) analysts are no longer alone in the quantity and variability of data they need to process to make decisions
2. www.netspective.com 2 @ShahidNShah Who is Shahid? • 25+ years of complex enterprise system architecture and engineering experience • 10+ years of high volume, high throughput data infrastructure architecture and operations • 7+ years of enterprise scale analytics • 5+ years as civilian researcher at Naval Surface Warfare Center (NSWC) • Industry knowledge: DoD, Civilian Public Sector, Healthcare, Medical Devices
3. @ShahidNShah www.netspective.com 3 What’s this talk about? ISR Landscape • Other industries’ data collection and processing has now matched ISR scale. • Other commercial sectors have similar needs for intelligence integration (e.g. marketing intel, sales intel, consumer intel, etc.) • The government & military are paying for cross-industry big data solutions. Key Takeaways • Data collection is mostly a solved problem and getting cheaper and easier • Data harmonization, homogenization, transport, ingestion, and integration are key areas of concern • Next generation big data tools in open source and commercial sectors can now be directly applied to next generation ISR needs with less customization than ever before • Government and military buyers have the procurement muscle to force vendors to adapt
4. @ShahidNShah www.netspective.com 4 Office of DNI IT Investment Strategy Lead/Influence • Exploitation – pattern discovery, context recognition, inferencing • Video/Motion Imagery – algorithm development, tagging & indexing streaming video • Large Data – both big static datasets and streaming media • Human Language Technology – spoken and written processing • Trust in software ,platforms & networks – support net-centric, multi-domain ops Adopt/Adapt • Computer Architecture / Cloud Computing – distributed processing and storage, quantum • Visualization – data and meta data visualization • Cognitive Systems – image and pattern analysis, integration of sound and touch, speech cognition • Knowledge Management – strategies and practices to gather, organize, and share organizational insights & experiences Source: Dawn Meyerriecks, Office of Director of National Intelligence, http://www.security-innovation.org/pdfs/Meyerriecks%20Presentation.pdf
5. @ShahidNShah www.netspective.com 5 How to buy next generation ISR tech The old way Identify problem Create RFPs Procure Custom Storage and Tools Deploy On Premises to Collect & Search Gain Value Continuously Customize Repeat process to Analyze & Visualize The new way Identify data Procure Existing Open Source/Commercial Deploy Cloud or Deploy On Premises Gain Value Provide Feedback to OSS Community or Customize Months / Years Weeks / Months
6. www.netspective.com 6 @ShahidNShah Data Comprehension is hard Data Comprehension and Explanation • OLD: custom reports • NEW: predictive analytics, CEP Data Integration and Analysis • OLD: all custom code • NEW: open source and other tools, non traditional ISR analysts coupled with traditional ISR analysis Data Aggregation • OLD: all custom code, all custom messages • NEW: ETL, ESB, and next generation data aggregation tool plus DDS / IoT messaging Data Collection • OLD: all custom sources, internally curated sources • NEW: some custom sources, externally curated sources, social data
7. @ShahidNShah www.netspective.com 7 NIST Big Data Reference Architecture Evaluation Presented as part of the NIST Big Data Working Group Meetings Source: http://bigdatawg.nist.gov
8. @ShahidNShah www.netspective.com 8 Apache Open Source Products Mapping Presented as part of the NIST Big Data Working Group Meetings http://bigdatawg.nist.gov/_uploadfiles/M0016_v1_4244171934.pdf
9. www.netspective.com 9 @ShahidNShah Think Big Analytics Reference Architecture Source: Think Big Analytics, http://thinkbiganalytics.com/leading_big_data_technologies/big-data-reference-architecture/
10. www.netspective.com 10 @ShahidNShah Oracle Big Data Reference Architecture Source: Oracle Corporation, http://www.oracle.com/technetwork/topics/entarch/articles/info-mgmt-big-data-ref-arch-1902853.pdf
11. www.netspective.com 11 @ShahidNShah Microsoft Big Data Reference Architecture Source: Microsoft
12. www.netspective.com 12 Microsoft Big Data Reference Architecture Source: Microsoft
13. @ShahidNShah www.netspective.com 13 Crate.io – Horizontally Scaled SQL & FTS Capabilities • Storing JSON structured documents via Lucene and ElasticSearch • Storing blobs • Finding documents via full text search • Real time data analytics using SQL • Making changes without re- doing everything Source: http://www.crate.io
14. @ShahidNShah www.netspective.com 14 Fast Data vs. Big Data Source: O’Reilly & Associates, Fast Data and The New Enterprise Data Architecture, 2014
15. www.netspective.com 15 @ShahidNShah BPM & Complex Event Processing (CEP) for ISR Source: TIBCO Software
16. www.netspective.com 16 @ShahidNShah Event Driven Architecture, Stream Processing Source: IBM
17. www.netspective.com 17 @ShahidNShah IoT Messaging, Data Distribution Service (DDS) Source: University of Granada
18. www.netspective.com 18 @ShahidNShah Tableau & Qlickview style visualizations
19. www.netspective.com 19 @ShahidNShah GeoCloud2 - Open source geospatial SaaS Source: http://www.mapcentia.com/en/geocloud/
20. @ShahidNShah www.netspective.com 20 Conclusion ISR Landscape • Other industries’ data collection and processing has now matched ISR scale. • Other commercial sectors have similar needs for intelligence integration (e.g. marketing intel, sales intel, consumer intel, etc.) • The government & military are paying for cross-industry big data solutions. Key Takeaways • Data collection is mostly a solved problem and getting cheaper and easier • Data harmonization, homogenization, transport, ingestion, and integration are key areas of concern • Next generation big data tools in open source and commercial sectors can now be directly applied to next generation ISR needs with less customization than ever before • Government and military buyers have the procurement muscle to force vendors to adapt
21. Conclusion and Questions @ShahidNShah shahid.shah@netspective.com www.ShahidShah.com