Advancing Affordable Energy With Advanced Energy Economy

Learn how the Advanced Energy Economy utilizes Elastic’s hosted Elasticsearch offering to power a scalable, secure, full-text search platform that makes more than 40M pages of energy policy data accessible to organizations looking to make energy cleaner, more affordable, and increasingly secure.

1. 1 1 Overview: •  Who We Are •  What We Do and Why It Matters •  How We Served Big Data and Lessons Learned •  Questions
2. 2 2 Who We Are
3. 3 National association of businesses making the global energy system more secure, clean, and affordable. Mission: Transform public policy to enable rapid growth of advanced energy companies.
4. 4 The Advanced Energy Economy includes many different types of companies and technologies. •  Building Efficiency •  Energy Efficiency •  Demand Response •  Electric Generation •  Solar •  Wind •  Grid Technologies •  Smart Grid •  EV Charging •  Transportation •  Plug-in / Hybrid Vehicles
5. 5 We have three distinct customer segments ranging from Fortune 100 companies to academic institutions. Commercial 1 Non-profit 2 Free 3
6. 6 We are a small team within a small non-profit. Eric Fitz Charlie Forcey Bradley Sheehan
7. 7 7 What We Do and Why It Matters
8. 8 PowerSuite: We make energy policy documents accessible and actionable across 50 states.   Identify and engage in policy issues   Grow the advanced energy industry Staying on top of energy policy is hard… PowerSuite makes it easy.
9. 9 Our platform has three layers. 50 states CA MA IL TX CT Collection Processing S3 Application 1 Middleware 2 Data Pipeline 3
10. 10 PowerSuite allows users to search, track, and collaborate on regulatory proceedings (“dockets”) from across the country. Staying on top of energy policy is hard… PowerSuite makes it easy. Search Collaborate Track Search
14. 14 Docket
15. 15 A “docket” is a collection of hundreds (or thousands) of documents submitted over many months. 270K Dockets 4M Documents 2TB of pdfs 45M Pages 90GB of text
16. 16 We have built the first national database of dockets and have more content than the English Wikipedia. 45M Pages 2TB of pdfs 90GB of text 270K Dockets 4M Documents How did we get here?
17. 17 17 How We Served Big Data and Lessons Learned
18. 18 At launch in 2014, our MVP full text search and primary database was powered by PostgreSQL. 50 states CA MA IL TX CT Collection Processing S3 Application 1 Middleware 2 Data Pipeline 3
19. 19 2015! Summer Our user and data growth trajectory indicated that we were going to scale out of PostgreSQL.   Index bloat   Search performance   Memory   Cost 2014 Summer
20. 20 20 Users don’t care about your pain.
21. 21 2015! Summer How do we scale our platform with limited resources, a looming cliff, and need to continue to deliver new features?   Index bloat   Search performance   Flexibility   Cost 2014 Summer 2015 Fall   Cluster Management?   Search Performance?   Flexibility?   Cost?
22. 22 22 Elastic cloud was the answer.
23. 23 With 45M documents, our sweet spot configuration was a cluster with 32GB memory, two nodes, and a tie-breaker.   Cluster Configuration:   One node per instance   16 Shards   32GB Memory   SSD Disk (tiebreaker) N0 N1 N2 (Found)
24. 24 Elastic Cloud allowed us to keep the lights on and significantly reduce our monthly costs. Cluster Management Search Performance & Flexibility Cost   Scale on demand   Push button upgrades   1.x 2.x   Simple plugin integration   Millisecond queries   Flexible tuning   Real-time indexing   Easy A/B testing   $1,000/month reduction   Opportunity Cost 1) No cluster management worries 2) All of Elastic that you know and love
25. 25 Elastic Cloud allowed us to keep the lights on and significantly reduce our monthly costs and… Cluster Management Search Performance & Flexibility Cost   Scale on demand   Push button upgrades   1.x 2.x   Simple plugin integration   Millisecond queries   Flexible tuning   Real-time indexing   Easy A/B testing   $1,000/month reduction   Opportunity Cost
26. 26 26 Deliver compelling user facing features.
27. 27 27 Summary and Questions
28. 28 Elastic is now our primary database. 50 states CA MA IL TX CT Collection Processing S3 Application 1 Middleware 2 Data Pipeline 3
29. 29 2016 Winter 2015! Summer Not only was the transition easy, but it has also unlocked a whole new class of features for PowerSuite.   Index bloat   Search performance   Memory   Cost   Advanced queries   Faceted search   Highlighting   Visualizations 2014 Summer 2015 Fall
30. 30 www.aee.net / @aeenet / Washington DC San Francisco Boston / powersuite.aee.net Questions? http://powersuite.aee.net Eric Fitz Senior Director Engineering and Product Development efitz@aee.net @ehfitz