Slide 1
Slide 1 text
How existing Big Data technologies are
impacting Next Generation ISR
Intelligence, Surveillance, and Reconnaisance (ISR) analysts are
no longer alone in the quantity and variability of data they
need to process to make decisions
Slide 2
Slide 2 text
www.netspective.com 2
@ShahidNShah
Who is Shahid?
• 25+ years of complex enterprise system architecture and engineering
experience
• 10+ years of high volume, high throughput data infrastructure architecture
and operations
• 7+ years of enterprise scale analytics
• 5+ years as civilian researcher at Naval Surface Warfare Center (NSWC)
• Industry knowledge: DoD, Civilian Public Sector, Healthcare, Medical
Devices
Slide 3
Slide 3 text
@ShahidNShah
www.netspective.com 3
What’s this talk about?
ISR Landscape
• Other industries’ data collection and
processing has now matched ISR
scale.
• Other commercial sectors have similar
needs for intelligence integration
(e.g. marketing intel, sales intel,
consumer intel, etc.)
• The government & military are paying
for cross-industry big data solutions.
Key Takeaways
• Data collection is mostly a solved
problem and getting cheaper and easier
• Data harmonization, homogenization,
transport, ingestion, and integration are
key areas of concern
• Next generation big data tools in open
source and commercial sectors can now
be directly applied to next generation ISR
needs with less customization than ever
before
• Government and military buyers have the
procurement muscle to force vendors to
adapt
Slide 4
Slide 4 text
@ShahidNShah
www.netspective.com 4
Office of DNI IT Investment Strategy
Lead/Influence
• Exploitation – pattern discovery, context
recognition, inferencing
• Video/Motion Imagery – algorithm
development, tagging & indexing
streaming video
• Large Data – both big static datasets and
streaming media
• Human Language Technology – spoken
and written processing
• Trust in software ,platforms & networks –
support net-centric, multi-domain ops
Adopt/Adapt
• Computer Architecture / Cloud
Computing – distributed processing and
storage, quantum
• Visualization – data and meta data
visualization
• Cognitive Systems – image and pattern
analysis, integration of sound and touch,
speech cognition
• Knowledge Management – strategies and
practices to gather, organize, and share
organizational insights & experiences
Source: Dawn Meyerriecks, Office of Director of National Intelligence, http://www.security-innovation.org/pdfs/Meyerriecks%20Presentation.pdf
Slide 5
Slide 5 text
@ShahidNShah
www.netspective.com 5
How to buy next generation ISR tech
The old way
Identify problem Create RFPs
Procure Custom
Storage and
Tools
Deploy On
Premises to
Collect & Search
Gain Value
Continuously
Customize
Repeat process
to Analyze &
Visualize
The new way
Identify data
Procure Existing Open
Source/Commercial
Deploy Cloud or Deploy
On Premises
Gain Value
Provide Feedback to OSS
Community or Customize
Months / Years
Weeks / Months
Slide 6
Slide 6 text
www.netspective.com 6
@ShahidNShah
Data Comprehension is hard
Data
Comprehension
and Explanation
• OLD: custom reports
• NEW: predictive analytics, CEP
Data Integration
and Analysis
• OLD: all custom code
• NEW: open source and other
tools, non traditional ISR
analysts coupled with
traditional ISR analysis
Data
Aggregation
• OLD: all custom code, all
custom messages
• NEW: ETL, ESB, and next
generation data aggregation
tool plus DDS / IoT
messaging
Data
Collection
• OLD: all custom sources,
internally curated sources
• NEW: some custom sources,
externally curated sources,
social data
Slide 7
Slide 7 text
@ShahidNShah
www.netspective.com 7
NIST Big Data Reference Architecture Evaluation
Presented as part of the NIST Big Data Working Group Meetings
Source: http://bigdatawg.nist.gov
Slide 8
Slide 8 text
@ShahidNShah
www.netspective.com 8
Apache Open Source Products Mapping
Presented as part of the NIST Big Data Working Group Meetings
http://bigdatawg.nist.gov/_uploadfiles/M0016_v1_4244171934.pdf
Slide 9
Slide 9 text
www.netspective.com 9
@ShahidNShah
Think Big Analytics Reference Architecture
Source: Think Big Analytics, http://thinkbiganalytics.com/leading_big_data_technologies/big-data-reference-architecture/
Slide 10
Slide 10 text
www.netspective.com 10
@ShahidNShah
Oracle Big Data Reference Architecture
Source: Oracle Corporation, http://www.oracle.com/technetwork/topics/entarch/articles/info-mgmt-big-data-ref-arch-1902853.pdf
Slide 11
Slide 11 text
www.netspective.com 11
@ShahidNShah
Microsoft Big Data Reference Architecture
Source: Microsoft
Slide 12
Slide 12 text
www.netspective.com 12
Microsoft Big Data Reference Architecture
Source: Microsoft
Slide 13
Slide 13 text
@ShahidNShah
www.netspective.com 13
Crate.io – Horizontally Scaled SQL & FTS
Capabilities
• Storing JSON structured
documents via Lucene and
ElasticSearch
• Storing blobs
• Finding documents via full text
search
• Real time data analytics using
SQL
• Making changes without re-
doing everything
Source: http://www.crate.io
Slide 14
Slide 14 text
@ShahidNShah
www.netspective.com 14
Fast Data vs. Big Data
Source: O’Reilly & Associates, Fast Data and The New Enterprise Data Architecture, 2014
Slide 15
Slide 15 text
www.netspective.com 15
@ShahidNShah
BPM & Complex Event Processing (CEP) for ISR
Source: TIBCO Software
Slide 16
Slide 16 text
www.netspective.com 16
@ShahidNShah
Event Driven Architecture, Stream Processing
Source: IBM
Slide 17
Slide 17 text
www.netspective.com 17
@ShahidNShah
IoT Messaging, Data Distribution Service (DDS)
Source: University of Granada
Slide 18
Slide 18 text
www.netspective.com 18
@ShahidNShah
Tableau & Qlickview style visualizations
Slide 19
Slide 19 text
www.netspective.com 19
@ShahidNShah
GeoCloud2 - Open source geospatial SaaS
Source: http://www.mapcentia.com/en/geocloud/
Slide 20
Slide 20 text
@ShahidNShah
www.netspective.com 20
Conclusion
ISR Landscape
• Other industries’ data collection and
processing has now matched ISR
scale.
• Other commercial sectors have similar
needs for intelligence integration
(e.g. marketing intel, sales intel,
consumer intel, etc.)
• The government & military are paying
for cross-industry big data solutions.
Key Takeaways
• Data collection is mostly a solved
problem and getting cheaper and easier
• Data harmonization, homogenization,
transport, ingestion, and integration are
key areas of concern
• Next generation big data tools in open
source and commercial sectors can now
be directly applied to next generation ISR
needs with less customization than ever
before
• Government and military buyers have the
procurement muscle to force vendors to
adapt