Slide 1

Slide 1 text

Jay Chin – [email protected] Principal Consultant, Excelian 3 November 2015 1   Grid Computing and Trade Analytics with Elastic

Slide 2

Slide 2 text

Excelian  Technical  Consulting   2 § Financial  Services  specialists     § Distributed  computing  specialists  since  2006     § Experts  in  niche  and  emerging  technologies   Thought  Leadership  &  Consul5ng   So7ware  Development  and  Engineering  Services   Run  Services   Our  services  

Slide 3

Slide 3 text

Financial  Services  –  Insatiable  appetite  for  Compute   • Algorithms  (Computers)  that  actually   do  the  trading     • Financial  modelling     • Huge  amounts  of  data  to  process   3 Source:  Information  Week,  Wall  Street  &  Technology   Source:  The  Telegraph  

Slide 4

Slide 4 text

What  do  compute  grids  look  like  ?   4 Typical  Numbers  For  A  Standard  Grid   -­‐  40k  cores/engines   -­‐  30m  tasks   -­‐  120  GB  of  Log  metrics   -­‐  60  –  80%  Average  Utilisation   -­‐  Data  retention  up  to  6  Months   h?ps://flic.kr/p/ydnEvw  

Slide 5

Slide 5 text

Grid  Maturity  in  Financial  Services   5 HPC Maturity Benchmark 2014 Tier  I  =  Tier  I  banks   Tier  II  =  Tier  II  banks   Point  =  point  solutions   used  only  for  a  specific   use  case  (e.g.  behind  a   software  package,   only  for  one  business   line…)   Maturity  Level  

Slide 6

Slide 6 text

Case  Study:  ELK  for  Enterprise  Grid  Reporting  Framework   • Enterprise  Grid  with  40,000  Cores  across  4  Data   centers  in  2  Countries     • Reporting  Dashboard  for  Grid  Metrics   • Scalable  up  to  100,000  cores  and  200  million  Grid   tasks  per  day   6 Goal:  Architect  an  Enterprise  Grid  and  design   a  Grid  metrics  reporting  framework  for  a  top-­‐ tier  investment  bank.  

Slide 7

Slide 7 text

The  Case  for  ELK   7 Features   Elas+cSearch   Intui5ve  Interface   Ease  of  Use   Security  Integra5on   Scalability   Support   Pricing   Features   Integra5on  with  Grid   Middleware  

Slide 8

Slide 8 text

Initial  Architecture  –  Single  cluster  across  2  regions   8 curl  -­‐XPUT  localhost:9200/GridA_metrics/_settings  -­‐d   '{  "index.routing.allocation.include.tag"  :  “region_A"  }'  

Slide 9

Slide 9 text

Architecture  (After  Consultation  with  Elastic  Platinum  Support)   9

Slide 10

Slide 10 text

Challenges 10 § Bespoke deployment due to security restrictions in Bank’s Datacentre. https://github.com/Excelian/ansible_fs_elkstack § Development of custom ETL to query Grid Metrics database and load them into ElasticSearch

Slide 11

Slide 11 text

More  ELK  Goodness   •  Bank  was  very  impressed  with  the   reporting  capabilities   •  Support  team  at  Elastic  was  also  superior   compared  to  some  of  the  big  vendors  we   were  dealing  with   11 AS A RESULT 1.  We  were  tasked  to  do  log  centralization  using   Logstash   2.  Explore  Watcher  for  monitoring  Grid  and   applications    

Slide 12

Slide 12 text

Feedback  from  Investment  Bank   • For  the  first  time  ever,   developers  were  able  to  view   Grid  metrics  correlate  them   with  logging  events  from  a   single  interface   • Application  teams  are   experimenting  with  Elastic  their   own  applications   • Developers  rethinking  logging   12

Slide 13

Slide 13 text

Key  Takeaways   • Lots  of  opportunities  and  interest  in  ElasticSearch  in   Financial  services   • Single  tool  to  do  log  analytics,  alerting,  events,   searching,  and  metrics   • Elastic  ticks  all  the  right  boxes  for  financial  services:   Security,  scalability,  support  SLAs,  etc.   • Elastic  Platinum  support  has  been  fantastic   • Advanced  Use  Cases  :  Fraud  Detection,  Trade   surveillance,  Market  Sentiment  Analysis     13

Slide 14

Slide 14 text

14 [email protected]   @excelian  

Slide 15

Slide 15 text

www.elas5c.co   15