Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Excelian's Grid Computing and Trade Analytics with Elastic

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
November 03, 2015

Excelian's Grid Computing and Trade Analytics with Elastic

Financial services have a huge appetite for compute cycles in order to meet the processing requirements for complex financial computation and data intensive processing. A common way to provide this is via grid computing at financial institutions. Jay provides details on how Elasticsearch was integrated into the grid computing stack at Excelian, a large investment bank, and describes experiences working with the product and the overall feedback received from the users.

Jay Chin | Elastic{ON} Tour | London


Elastic Co

November 03, 2015

More Decks by Elastic Co

Other Decks in Technology


  1. Jay Chin – jay.chin@Excelian.com Principal Consultant, Excelian 3 November 2015

    1   Grid Computing and Trade Analytics with Elastic
  2. Excelian  Technical  Consulting   2 § Financial  Services  specialists    

    § Distributed  computing  specialists  since  2006     § Experts  in  niche  and  emerging  technologies   Thought  Leadership  &  Consul5ng   So7ware  Development  and  Engineering  Services   Run  Services   Our  services  
  3. Financial  Services  –  Insatiable  appetite  for  Compute   • Algorithms  (Computers)

     that  actually   do  the  trading     • Financial  modelling     • Huge  amounts  of  data  to  process   3 Source:  Information  Week,  Wall  Street  &  Technology   Source:  The  Telegraph  
  4. What  do  compute  grids  look  like  ?   4 Typical

     Numbers  For  A  Standard  Grid   -­‐  40k  cores/engines   -­‐  30m  tasks   -­‐  120  GB  of  Log  metrics   -­‐  60  –  80%  Average  Utilisation   -­‐  Data  retention  up  to  6  Months   h?ps://flic.kr/p/ydnEvw  
  5. Grid  Maturity  in  Financial  Services   5 HPC Maturity Benchmark

    2014 Tier  I  =  Tier  I  banks   Tier  II  =  Tier  II  banks   Point  =  point  solutions   used  only  for  a  specific   use  case  (e.g.  behind  a   software  package,   only  for  one  business   line…)   Maturity  Level  
  6. Case  Study:  ELK  for  Enterprise  Grid  Reporting  Framework   • Enterprise

     Grid  with  40,000  Cores  across  4  Data   centers  in  2  Countries     • Reporting  Dashboard  for  Grid  Metrics   • Scalable  up  to  100,000  cores  and  200  million  Grid   tasks  per  day   6 Goal:  Architect  an  Enterprise  Grid  and  design   a  Grid  metrics  reporting  framework  for  a  top-­‐ tier  investment  bank.  
  7. The  Case  for  ELK   7 Features   Elas+cSearch  

    Intui5ve  Interface   Ease  of  Use   Security  Integra5on   Scalability   Support   Pricing   Features   Integra5on  with  Grid   Middleware  
  8. Initial  Architecture  –  Single  cluster  across  2  regions   8

    curl  -­‐XPUT  localhost:9200/GridA_metrics/_settings  -­‐d   '{  "index.routing.allocation.include.tag"  :  “region_A"  }'  
  9. Architecture  (After  Consultation  with  Elastic  Platinum  Support)   9

  10. Challenges 10 § Bespoke deployment due to security restrictions in Bank’s

    Datacentre. https://github.com/Excelian/ansible_fs_elkstack § Development of custom ETL to query Grid Metrics database and load them into ElasticSearch
  11. More  ELK  Goodness   •  Bank  was  very  impressed  with

     the   reporting  capabilities   •  Support  team  at  Elastic  was  also  superior   compared  to  some  of  the  big  vendors  we   were  dealing  with   11 AS A RESULT 1.  We  were  tasked  to  do  log  centralization  using   Logstash   2.  Explore  Watcher  for  monitoring  Grid  and   applications    
  12. Feedback  from  Investment  Bank   • For  the  first  time  ever,

      developers  were  able  to  view   Grid  metrics  correlate  them   with  logging  events  from  a   single  interface   • Application  teams  are   experimenting  with  Elastic  their   own  applications   • Developers  rethinking  logging   12
  13. Key  Takeaways   • Lots  of  opportunities  and  interest  in  ElasticSearch

     in   Financial  services   • Single  tool  to  do  log  analytics,  alerting,  events,   searching,  and  metrics   • Elastic  ticks  all  the  right  boxes  for  financial  services:   Security,  scalability,  support  SLAs,  etc.   • Elastic  Platinum  support  has  been  fantastic   • Advanced  Use  Cases  :  Fraud  Detection,  Trade   surveillance,  Market  Sentiment  Analysis     13
  14. 14 jay.chin@excelian.com   @excelian  

  15. www.elas5c.co   15