Excelian's Grid Computing and Trade Analytics with Elastic

Financial services have a huge appetite for compute cycles in order to meet the processing requirements for complex financial computation and data intensive processing. A common way to provide this is via grid computing at financial institutions. Jay provides details on how Elasticsearch was integrated into the grid computing stack at Excelian, a large investment bank, and describes experiences working with the product and the overall feedback received from the users.

Jay Chin | Elastic{ON} Tour | London

Transcript

1. Jay Chin – jay.chin@Excelian.com
   Principal Consultant, Excelian
   3 November 2015
   1  
   Grid Computing and Trade Analytics
   with Elastic
   

   View Slide

2. Excelian  Technical  Consulting  
   2
   § Financial  Services  specialists  
    
   § Distributed  computing  specialists  since  2006  
    
   § Experts  in  niche  and  emerging  technologies  
   Thought  Leadership  &  Consul5ng  
   So7ware  Development  and  Engineering  Services  
   Run  Services  
   Our  services  
   

   View Slide

3. Financial  Services  –  Insatiable  appetite  for  Compute  
   • Algorithms  (Computers)  that  actually  
   do  the  trading    
   • Financial  modelling    
   • Huge  amounts  of  data  to  process  
   3
   Source:  Information  Week,  Wall  Street  &  Technology   Source:  The  Telegraph  
   

   View Slide

4. What  do  compute  grids  look  like  ?  
   4
   Typical  Numbers  For  A  Standard  Grid  
   -­‐  40k  cores/engines  
   -­‐  30m  tasks  
   -­‐  120  GB  of  Log  metrics  
   -­‐  60  –  80%  Average  Utilisation  
   -­‐  Data  retention  up  to  6  Months  
   h?ps://flic.kr/p/ydnEvw  
   

   View Slide

5. Grid  Maturity  in  Financial  Services  
   5
   HPC Maturity
   Benchmark
   2014
   Tier  I  =  Tier  I  banks  
   Tier  II  =  Tier  II  banks  
   Point  =  point  solutions  
   used  only  for  a  specific  
   use  case  (e.g.  behind  a  
   software  package,  
   only  for  one  business  
   line…)  
   Maturity  Level  
   

   View Slide

6. Case  Study:  ELK  for  Enterprise  Grid  Reporting  Framework  
   • Enterprise  Grid  with  40,000  Cores  across  4  Data  
   centers  in  2  Countries    
   • Reporting  Dashboard  for  Grid  Metrics  
   • Scalable  up  to  100,000  cores  and  200  million  Grid  
   tasks  per  day  
   6
   Goal:  Architect  an  Enterprise  Grid  and  design  
   a  Grid  metrics  reporting  framework  for  a  top-­‐
   tier  investment  bank.  
   

   View Slide

7. The  Case  for  ELK  
   7
   Features   Elas+cSearch  
   Intui5ve  Interface  
   Ease  of  Use  
   Security  Integra5on  
   Scalability  
   Support  
   Pricing  
   Features  
   Integra5on  with  Grid  
   Middleware  
   

   View Slide

8. Initial  Architecture  –  Single  cluster  across  2  regions  
   8
   curl  -­‐XPUT  localhost:9200/GridA_metrics/_settings  -­‐d  
   '{  "index.routing.allocation.include.tag"  :  “region_A"  }'  
   

   View Slide

9. Architecture  (After  Consultation  with  Elastic  Platinum  Support)  
   9
   

   View Slide

10. Challenges
    10
    § Bespoke deployment due to security restrictions in
    Bank’s Datacentre.
    https://github.com/Excelian/ansible_fs_elkstack
    § Development of custom ETL to query Grid Metrics
    database and load them into ElasticSearch
    

    View Slide

11. More  ELK  Goodness  
    •  Bank  was  very  impressed  with  the  
    reporting  capabilities  
    •  Support  team  at  Elastic  was  also  superior  
    compared  to  some  of  the  big  vendors  we  
    were  dealing  with  
    11
    AS A RESULT
    1.  We  were  tasked  to  do  log  centralization  using  
    Logstash  
    2.  Explore  Watcher  for  monitoring  Grid  and  
    applications    
    

    View Slide

12. Feedback  from  Investment  Bank  
    • For  the  first  time  ever,  
    developers  were  able  to  view  
    Grid  metrics  correlate  them  
    with  logging  events  from  a  
    single  interface  
    • Application  teams  are  
    experimenting  with  Elastic  their  
    own  applications  
    • Developers  rethinking  logging  
    12
    

    View Slide

13. Key  Takeaways  
    • Lots  of  opportunities  and  interest  in  ElasticSearch  in  
    Financial  services  
    • Single  tool  to  do  log  analytics,  alerting,  events,  
    searching,  and  metrics  
    • Elastic  ticks  all  the  right  boxes  for  financial  services:  
    Security,  scalability,  support  SLAs,  etc.  
    • Elastic  Platinum  support  has  been  fantastic  
    • Advanced  Use  Cases  :  Fraud  Detection,  Trade  
    surveillance,  Market  Sentiment  Analysis  
     
    13
    

    View Slide

14. 14
    jay.chin@excelian.com  
    @excelian  
    

    View Slide

15. www.elas5c.co  
    15  
    

    View Slide