Unlocking Interplanetary Datasets with Real-Time Search

This talk was presented at the inaugural Elastic{ON} conference, http://elasticon.com

Session Abstract:
Hear how the Jet Propulsion Laboratory is using Elasticsearch to power flight telemetry and system dashboards from the Curiosity Mars rover and SMAP, as well as its internal search engine including a powerful video and people search.

Transcript

1. { } 29 Unlocking Interplanetary Datasets with Realtime Search Ricky Ma and Dan Isla
2. { } 30 Ricky Ma Chief of Application Architecture Dan Isla Systems Engineer and Data Scientist Who We Are
3. About JPL
4. { } 32 We are one of the ten NASA field centers
5. { } 33 we have about 5000 employees
6. { } 34 located in Pasadena California...
7. { } 35 and we make stuff
8. { } 36 Spacecraft (some of the many…) 1st US Satellite 89 - Study Jupiter and its moons 77 - Most distant human-made object
9. { } 37 First US Satellite
10. { } 38 Furthest Man Made Object
12. Spacecraft Designs People / Collaboration Documents and Data =
14. Custom Admin panels
15. Adaptive Learning
18. http://space.io9.com/this-is-every-place-in-the-solar-system-ever-occupied-e-1645507693
19. http://www.nasa.gov/sites/default/files/14-198.png
21. Legacy SQL Ground System Excel/Matlab PowerPoint E-Mail
24. Improved Min/Max Stats metric loses exact time of min and max value. Top Hits nested aggregation, using _source to extract time.
25. { } 53 Improved Min/Max Stats metric loses exact time of min and max value. Top Hits nested aggregation, using _source to extract time.
26. { } 54 Data Pipeline and Indexing ETL
27. { } 55
29. { } 57 Earth Science Earth Science: SMAP • 30x more telemetry than Mars mission. • 80 million telemetry points per day. • 3.8 billion docs generated during test and launch campaign. • 6 node i2.xlarge cluster, 1.5+ TB ingested. • Sub-second query time
30. • Mesos to orchestrate multi cluster deployments of Elasticsearch • Docker containers for Elasticsearch Nodes • Hybrid Cloud infrastructure with AWS GovCloud and Nebula private cloud. Deployment
31. _search?q=life:universe
32. { } 60 Thanks for listening! Ricky Ma and Dan Isla