How existing Big Data technologies are impacting Next Generation ISR

Subtitle: Intelligence, Surveillance, and Reconnaisance (ISR) analysts are no longer alone in the quantity and variability of data they need to process to make decisions

This talk was given at the "Next Generation ISR" Summit on Thursday, December 11.

Background / Landscape:

* Other industries’ data collection and processing has now matched ISR scale.
* Other commercial sectors have similar needs for intelligence integration (e.g. marketing intel, sales intel, consumer intel, etc.)
* The government & military are paying for cross-industry big data solutions.

Key Takeaways:
* Data collection is mostly a solved problem and getting cheaper and easier
* Data harmonization, homogenization, transport, ingestion, and integration are key areas of concern
* Next generation big data tools in open source and commercial sectors can now be directly applied to next generation ISR needs with less customization than ever before
* Government and military buyers have the procurement muscle to force vendors to adapt

Transcript

1. How existing Big Data technologies are
   impacting Next Generation ISR
   Intelligence, Surveillance, and Reconnaisance (ISR) analysts are
   no longer alone in the quantity and variability of data they
   need to process to make decisions
   

   View Slide

2. www.netspective.com 2
   @ShahidNShah
   Who is Shahid?
   • 25+ years of complex enterprise system architecture and engineering
   experience
   • 10+ years of high volume, high throughput data infrastructure architecture
   and operations
   • 7+ years of enterprise scale analytics
   • 5+ years as civilian researcher at Naval Surface Warfare Center (NSWC)
   • Industry knowledge: DoD, Civilian Public Sector, Healthcare, Medical
   Devices
   

   View Slide

3. @ShahidNShah
   www.netspective.com 3
   What’s this talk about?
   ISR Landscape
   • Other industries’ data collection and
   processing has now matched ISR
   scale.
   • Other commercial sectors have similar
   needs for intelligence integration
   (e.g. marketing intel, sales intel,
   consumer intel, etc.)
   • The government & military are paying
   for cross-industry big data solutions.
   Key Takeaways
   • Data collection is mostly a solved
   problem and getting cheaper and easier
   • Data harmonization, homogenization,
   transport, ingestion, and integration are
   key areas of concern
   • Next generation big data tools in open
   source and commercial sectors can now
   be directly applied to next generation ISR
   needs with less customization than ever
   before
   • Government and military buyers have the
   procurement muscle to force vendors to
   adapt
   

   View Slide

4. @ShahidNShah
   www.netspective.com 4
   Office of DNI IT Investment Strategy
   Lead/Influence
   • Exploitation – pattern discovery, context
   recognition, inferencing
   • Video/Motion Imagery – algorithm
   development, tagging & indexing
   streaming video
   • Large Data – both big static datasets and
   streaming media
   • Human Language Technology – spoken
   and written processing
   • Trust in software ,platforms & networks –
   support net-centric, multi-domain ops
   Adopt/Adapt
   • Computer Architecture / Cloud
   Computing – distributed processing and
   storage, quantum
   • Visualization – data and meta data
   visualization
   • Cognitive Systems – image and pattern
   analysis, integration of sound and touch,
   speech cognition
   • Knowledge Management – strategies and
   practices to gather, organize, and share
   organizational insights & experiences
   Source: Dawn Meyerriecks, Office of Director of National Intelligence, http://www.security-innovation.org/pdfs/Meyerriecks%20Presentation.pdf
   

   View Slide

5. @ShahidNShah
   www.netspective.com 5
   How to buy next generation ISR tech
   The old way
   Identify problem Create RFPs
   Procure Custom
   Storage and
   Tools
   Deploy On
   Premises to
   Collect & Search
   Gain Value
   Continuously
   Customize
   Repeat process
   to Analyze &
   Visualize
   The new way
   Identify data
   Procure Existing Open
   Source/Commercial
   Deploy Cloud or Deploy
   On Premises
   Gain Value
   Provide Feedback to OSS
   Community or Customize
   Months / Years
   Weeks / Months
   

   View Slide

6. www.netspective.com 6
   @ShahidNShah
   Data Comprehension is hard
   Data
   Comprehension
   and Explanation
   • OLD: custom reports
   • NEW: predictive analytics, CEP
   Data Integration
   and Analysis
   • OLD: all custom code
   • NEW: open source and other
   tools, non traditional ISR
   analysts coupled with
   traditional ISR analysis
   Data
   Aggregation
   • OLD: all custom code, all
   custom messages
   • NEW: ETL, ESB, and next
   generation data aggregation
   tool plus DDS / IoT
   messaging
   Data
   Collection
   • OLD: all custom sources,
   internally curated sources
   • NEW: some custom sources,
   externally curated sources,
   social data
   

   View Slide

7. @ShahidNShah
   www.netspective.com 7
   NIST Big Data Reference Architecture Evaluation
   Presented as part of the NIST Big Data Working Group Meetings
   Source: http://bigdatawg.nist.gov
   

   View Slide

8. @ShahidNShah
   www.netspective.com 8
   Apache Open Source Products Mapping
   Presented as part of the NIST Big Data Working Group Meetings
   http://bigdatawg.nist.gov/_uploadfiles/M0016_v1_4244171934.pdf
   

   View Slide

9. www.netspective.com 9
   @ShahidNShah
   Think Big Analytics Reference Architecture
   Source: Think Big Analytics, http://thinkbiganalytics.com/leading_big_data_technologies/big-data-reference-architecture/
   

   View Slide

10. www.netspective.com 10
    @ShahidNShah
    Oracle Big Data Reference Architecture
    Source: Oracle Corporation, http://www.oracle.com/technetwork/topics/entarch/articles/info-mgmt-big-data-ref-arch-1902853.pdf
    

    View Slide

11. www.netspective.com 11
    @ShahidNShah
    Microsoft Big Data Reference Architecture
    Source: Microsoft
    

    View Slide

12. www.netspective.com 12
    Microsoft Big Data Reference Architecture
    Source: Microsoft
    

    View Slide

13. @ShahidNShah
    www.netspective.com 13
    Crate.io – Horizontally Scaled SQL & FTS
    Capabilities
    • Storing JSON structured
    documents via Lucene and
    ElasticSearch
    • Storing blobs
    • Finding documents via full text
    search
    • Real time data analytics using
    SQL
    • Making changes without re-
    doing everything
    Source: http://www.crate.io
    

    View Slide

14. @ShahidNShah
    www.netspective.com 14
    Fast Data vs. Big Data
    Source: O’Reilly & Associates, Fast Data and The New Enterprise Data Architecture, 2014
    

    View Slide

15. www.netspective.com 15
    @ShahidNShah
    BPM & Complex Event Processing (CEP) for ISR
    Source: TIBCO Software
    

    View Slide

16. www.netspective.com 16
    @ShahidNShah
    Event Driven Architecture, Stream Processing
    Source: IBM
    

    View Slide

17. www.netspective.com 17
    @ShahidNShah
    IoT Messaging, Data Distribution Service (DDS)
    Source: University of Granada
    

    View Slide

18. www.netspective.com 18
    @ShahidNShah
    Tableau & Qlickview style visualizations
    

    View Slide

19. www.netspective.com 19
    @ShahidNShah
    GeoCloud2 - Open source geospatial SaaS
    Source: http://www.mapcentia.com/en/geocloud/
    

    View Slide

20. @ShahidNShah
    www.netspective.com 20
    Conclusion
    ISR Landscape
    • Other industries’ data collection and
    processing has now matched ISR
    scale.
    • Other commercial sectors have similar
    needs for intelligence integration
    (e.g. marketing intel, sales intel,
    consumer intel, etc.)
    • The government & military are paying
    for cross-industry big data solutions.
    Key Takeaways
    • Data collection is mostly a solved
    problem and getting cheaper and easier
    • Data harmonization, homogenization,
    transport, ingestion, and integration are
    key areas of concern
    • Next generation big data tools in open
    source and commercial sectors can now
    be directly applied to next generation ISR
    needs with less customization than ever
    before
    • Government and military buyers have the
    procurement muscle to force vendors to
    adapt
    

    View Slide

21. Conclusion and Questions
    @ShahidNShah shahid.shah@netspective.com
    www.ShahidShah.com
    

    View Slide