Excelian's Grid Computing and Trade Analytics with Elastic

Slide 1

Slide 1 text

Jay Chin – [email protected] Principal Consultant, Excelian 3 November 2015 1 Grid Computing and Trade Analytics with Elastic

Slide 2

Slide 2 text

Excelian Technical Consulting 2 § Financial Services specialists § Distributed computing specialists since 2006 § Experts in niche and emerging technologies Thought Leadership & Consul5ng So7ware Development and Engineering Services Run Services Our services

Slide 3

Slide 3 text

Financial Services – Insatiable appetite for Compute • Algorithms (Computers) that actually do the trading • Financial modelling • Huge amounts of data to process 3 Source: Information Week, Wall Street & Technology Source: The Telegraph

Slide 4

Slide 4 text

What do compute grids look like ? 4 Typical Numbers For A Standard Grid -‐  40k cores/engines -‐  30m tasks -‐  120 GB of Log metrics -‐  60 – 80% Average Utilisation -‐  Data retention up to 6 Months h?ps://ﬂic.kr/p/ydnEvw

Slide 5

Slide 5 text

Grid Maturity in Financial Services 5 HPC Maturity Benchmark 2014 Tier I = Tier I banks Tier II = Tier II banks Point = point solutions used only for a speciﬁc use case (e.g. behind a software package, only for one business line…) Maturity Level

Slide 6

Slide 6 text

Case Study: ELK for Enterprise Grid Reporting Framework • Enterprise Grid with 40,000 Cores across 4 Data centers in 2 Countries • Reporting Dashboard for Grid Metrics • Scalable up to 100,000 cores and 200 million Grid tasks per day 6 Goal: Architect an Enterprise Grid and design a Grid metrics reporting framework for a top-‐ tier investment bank.

Slide 7

Slide 7 text

The Case for ELK 7 Features Elas+cSearch Intui5ve Interface Ease of Use Security Integra5on Scalability Support Pricing Features Integra5on with Grid Middleware

Slide 8

Slide 8 text

Initial Architecture – Single cluster across 2 regions 8 curl -‐XPUT localhost:9200/GridA_metrics/_settings -‐d '{ "index.routing.allocation.include.tag" : “region_A" }'

Slide 9

Slide 9 text

Architecture (After Consultation with Elastic Platinum Support) 9

Slide 10

Slide 10 text

Challenges 10 § Bespoke deployment due to security restrictions in Bank’s Datacentre. https://github.com/Excelian/ansible_fs_elkstack § Development of custom ETL to query Grid Metrics database and load them into ElasticSearch

Slide 11

Slide 11 text

More ELK Goodness •  Bank was very impressed with the reporting capabilities •  Support team at Elastic was also superior compared to some of the big vendors we were dealing with 11 AS A RESULT 1. We were tasked to do log centralization using Logstash 2. Explore Watcher for monitoring Grid and applications

Slide 12

Slide 12 text

Feedback from Investment Bank • For the ﬁrst time ever, developers were able to view Grid metrics correlate them with logging events from a single interface • Application teams are experimenting with Elastic their own applications • Developers rethinking logging 12

Slide 13

Slide 13 text

Key Takeaways • Lots of opportunities and interest in ElasticSearch in Financial services • Single tool to do log analytics, alerting, events, searching, and metrics • Elastic ticks all the right boxes for ﬁnancial services: Security, scalability, support SLAs, etc. • Elastic Platinum support has been fantastic • Advanced Use Cases : Fraud Detection, Trade surveillance, Market Sentiment Analysis 13