Excelian's Grid Computing and Trade Analytics with Elastic

Financial services have a huge appetite for compute cycles in order to meet the processing requirements for complex financial computation and data intensive processing. A common way to provide this is via grid computing at financial institutions. Jay provides details on how Elasticsearch was integrated into the grid computing stack at Excelian, a large investment bank, and describes experiences working with the product and the overall feedback received from the users.

Jay Chin | Elastic{ON} Tour | London

Transcript

1. Jay Chin – jay.chin@Excelian.com Principal Consultant, Excelian 3 November 2015 1   Grid Computing and Trade Analytics with Elastic
2. Excelian  Technical  Consulting   2 § Financial  Services  specialists     § Distributed  computing  specialists  since  2006     § Experts  in  niche  and  emerging  technologies   Thought  Leadership  &  Consul5ng   So7ware  Development  and  Engineering  Services   Run  Services   Our  services  
3. Financial  Services  –  Insatiable  appetite  for  Compute   • Algorithms  (Computers)  that  actually   do  the  trading     • Financial  modelling     • Huge  amounts  of  data  to  process   3 Source:  Information  Week,  Wall  Street  &  Technology   Source:  The  Telegraph  
4. What  do  compute  grids  look  like  ?   4 Typical  Numbers  For  A  Standard  Grid   -­‐  40k  cores/engines   -­‐  30m  tasks   -­‐  120  GB  of  Log  metrics   -­‐  60  –  80%  Average  Utilisation   -­‐  Data  retention  up  to  6  Months   h?ps://flic.kr/p/ydnEvw  
5. Grid  Maturity  in  Financial  Services   5 HPC Maturity Benchmark 2014 Tier  I  =  Tier  I  banks   Tier  II  =  Tier  II  banks   Point  =  point  solutions   used  only  for  a  specific   use  case  (e.g.  behind  a   software  package,   only  for  one  business   line…)   Maturity  Level  
6. Case  Study:  ELK  for  Enterprise  Grid  Reporting  Framework   • Enterprise  Grid  with  40,000  Cores  across  4  Data   centers  in  2  Countries     • Reporting  Dashboard  for  Grid  Metrics   • Scalable  up  to  100,000  cores  and  200  million  Grid   tasks  per  day   6 Goal:  Architect  an  Enterprise  Grid  and  design   a  Grid  metrics  reporting  framework  for  a  top-­‐ tier  investment  bank.  
7. The  Case  for  ELK   7 Features   Elas+cSearch   Intui5ve  Interface   Ease  of  Use   Security  Integra5on   Scalability   Support   Pricing   Features   Integra5on  with  Grid   Middleware  
8. Initial  Architecture  –  Single  cluster  across  2  regions   8 curl  -­‐XPUT  localhost:9200/GridA_metrics/_settings  -­‐d   '{  "index.routing.allocation.include.tag"  :  “region_A"  }'  
9. Architecture  (After  Consultation  with  Elastic  Platinum  Support)   9
10. Challenges 10 § Bespoke deployment due to security restrictions in Bank’s Datacentre. https://github.com/Excelian/ansible_fs_elkstack § Development of custom ETL to query Grid Metrics database and load them into ElasticSearch
11. More  ELK  Goodness   •  Bank  was  very  impressed  with  the   reporting  capabilities   •  Support  team  at  Elastic  was  also  superior   compared  to  some  of  the  big  vendors  we   were  dealing  with   11 AS A RESULT 1.  We  were  tasked  to  do  log  centralization  using   Logstash   2.  Explore  Watcher  for  monitoring  Grid  and   applications    
12. Feedback  from  Investment  Bank   • For  the  first  time  ever,   developers  were  able  to  view   Grid  metrics  correlate  them   with  logging  events  from  a   single  interface   • Application  teams  are   experimenting  with  Elastic  their   own  applications   • Developers  rethinking  logging   12
13. Key  Takeaways   • Lots  of  opportunities  and  interest  in  ElasticSearch  in   Financial  services   • Single  tool  to  do  log  analytics,  alerting,  events,   searching,  and  metrics   • Elastic  ticks  all  the  right  boxes  for  financial  services:   Security,  scalability,  support  SLAs,  etc.   • Elastic  Platinum  support  has  been  fantastic   • Advanced  Use  Cases  :  Fraud  Detection,  Trade   surveillance,  Market  Sentiment  Analysis     13
14. 14 jay.chin@excelian.com   @excelian  
15. www.elas5c.co   15