SysFera-DS @ ROMEO-CRIHAN

Presentation made during the Bull User Group (BUX), on March 19th, 2013.
Given by B. Depardon

Transcript

1. SysFera-­‐DS   @   ROMEO  –  CRIHAN   BUX,  March  2013   Benjamin  Depardon   CTO@SysFera   ©  SysFera     HPC  and  SaaS  made  easy  
2. SysFera   •  2001-­‐2010   – Laboratoire  de  l’InformaIque  du  parallélisme   (ENS,  INRIA),  Project-­‐Team  Graal/Avalon   – Décrypthon  Project  with  AFM,  IBM   •  2010-­‐2013   – SysFera   – Co-­‐developing  with  EDF  R&D   – ParIcipate  in  compuIng  plaZorm  projects  (IRT,   ANR,  FP7,  Mésocentres,  Décrypthon  3)   ©  SysFera     SysFera-­‐DS  
3. SysFera  –  Vision   •  Ease  of  use  of  HPC  resources   •  Researchers/engineers  must  concentrate  on   their  field  of  experIse,  not  on  computer   – Ease  of  use   – Transparent  access  to  resources   – Federate  access  to  resources   ©  SysFera     SysFera-­‐DS  
4. ©  SysFera     SysFera-­‐DS   HTML5 Browser Mono-­‐cluster   Mul2-­‐cluster  
5. ROMEO  –  CRIHAN   ©  SysFera     SysFera-­‐DS   !!" LoadLeveler PBS Pro
6. ROMEO  –  CRIHAN   ©  SysFera     SysFera-­‐DS   LoadLeveler PBS Pro
7. WEBBOARD   ©  SysFera     SysFera-­‐DS  
8. WebBoard  -­‐  PresentaIon   •  CollaboraIve  tool  dedicated  to  HPC   –  Projects  management   –  Access  rights  management   –  ApplicaIons  management   –  AccounIng,  staIsIcs  and  acIvity   –  Relies  on  an  infrastructure  manager  (e.g.,  Vishnu,  BS)   •  Flexible:  easily  adaptable   –  Remote  visualizaIon  plugin   –  File  management  plugin   –  Graphical  UI  is  adaptable   ©  SysFera     SysFera-­‐DS  
9. WebBoard   ©  SysFera     SysFera-­‐DS  
10. WebBoard  –  ApplicaIons   ©  SysFera     SysFera-­‐DS  
11. WebBoard  –  GesIon  de  fichiers   ©  SysFera     SysFera-­‐DS  
12. WebBoard  –  Remote  VisualizaIon   ©  SysFera     SysFera-­‐DS  
13. Remote  visualizaIon   ©  SysFera     SysFera-­‐DS  
14. WebBoard  -­‐  Deployment   •  Easy  installaIon   –  Standard  Tomcat  and  a  war   –  ConnecIon  with  main  DB  (Postgres,  MySQL)   –  Plugged  to  the  infrastructure  manager  thanks  to  a   message  bus  (RabbitMQ)   •  ConfiguraIon  with  1  file   –  Host/Port/type  for  DB  access   –  Mail  server  configuraIon  for  noIficaIons   –  Host/Port  for  RabbitMQ   –  Plugins  acIvaIon/deacIvaIon   ©  SysFera     SysFera-­‐DS  
15. WebBoard  –  Advantages   •  Remote  access  to  distant  machines  through  a   simple  web  browser  on  the  client  side   •  Mutualize  computaIonal  resources  usage   •  Batch  computaIon  or  visualizaIon   •  ReservaIon  of  remote  visualizaIon  sessions  in   advance   •  Projects,  access  rights,  applicaIons  and  files   management   ©  SysFera     SysFera-­‐DS  
16. VISHNU  MIDDLEWARE   ©  SysFera     SysFera-­‐DS  
17. Vishnu   ©  SysFera     SysFera-­‐DS   h@p://github.com/sysfera/vishnu  
18. Vishnu  –  Components   ©  SysFera     SysFera-­‐DS   Java API Python API Command line interface Vishnu API (C++) UMS client IMS client FMS client TMS client UMS server IMS server FMS server TMS server ZMQ Database SCP/RSYNC Batch schedulers Communication bus Vishnu interface software component Vishnu internal software component External components Client side Server side Dispatcher
19. Vishnu  -­‐  Deployment   •  Distributed  soluIon   –  ConnecIon  with  main  DB  (Postgres,  MySQL)   –  Daemons  to  deploy  on  each  cluster’s  gateway  (TMS,  IMS)   –  “Central”  daemons  (UMS,  FMS)   –  OpIonally:  dispatcher  (proxy)   •  ConfiguraIon  with  1  file   –  Host/Port/type  for  DB   –  Mail  server  configuraIon  for  noIficaIons   –  Host/Port  for  ZeroMQ   –  Batch  scheduler  type   •  Works  in  userspace   ©  SysFera     SysFera-­‐DS  
20. TMS  –  Generic  script   ©  SysFera     Batch  scheduler  specific  parameters   Batch  scheduler  generic  parameters   Generic  variables   Script’s  parameters:  strings  or  files   SysFera-­‐DS  
21. Local  accounts  management   ©  SysFera     Site 1 /home/bobby Site 2 /home/bob1 Site 3 /home/bob             SeD SysFera-DS End-User Computer    
    SeD SeD Identification DB (1) Open session (3) Send requests ... (6) Close session (4) Use global ID (5) ssh as Bobby (5) ssh as Bob1 (5) ssh as Bob (2) Retreive identification credentials LDAP1,2...n SysFera-­‐DS  
22. Vishnu  –  Advantages   •  Mutualize  compuIng  resources   •  Non  intrusive  /  non  exclusive   •  Remote  access  to  distant  resources  through   various  API  C++/Python/Java,  and  CLI   •  Abstract  layer  to  different  batch  schedulers   –  SLURM,  SGE,  Torque,  PBS  Pro,  LoadLeveler,  LSF   –  Cloud:  OpenStack  (DeltaCloud)   •  End-­‐to-­‐end  users’  idenIty  management   •  Distributed  and  fault  tolerant  architecture   ©  SysFera     SysFera-­‐DS  
23. CURRENT  DEPLOYMENTS   ©  SysFera     SysFera-­‐DS  
24. ROMEO-­‐CRIHAN   ©  SysFera     SysFera-­‐DS   PBS Pro LoadLeveler !"#!$%&'%$(!)&*% +",-$).),/01 '",2$3&4)$25%0$63&*% +",-$).),/01 ''%5603,&4)$25%0$63&*% SLURM '",2$3578 User1 HTML5 Browser User2 SSL Internet + Firewalls 9$34!"$:!;,).),/0 <,6=4#0 > ?!#! @,/03
25. Other  plaZorms   •  EDF  R&D   •  IRT  BioAster  (CC-­‐IN2P3  –  Pasteur  InsItute)   •  CINES   •  Decrypthon  3   ©  SysFera     SysFera-­‐DS  
26. SALOME  &  Vishnu   •  Open-­‐source  mulI-­‐physics  plaZorm   •  Through  its  JOBMANAGER  module,  SALOME   can:  submit,  follow,  cancel  jobs,  transfer  files   before  and  aler  jobs’  execuIon   •  Benefits  of  using  Vishnu   – Access  to  a  new  machine  can  be  done  without  any   modificaIon  in  SALOME,  whatever  the  BS   – SALOME’s  jobs  can  be  managed  through  several   interfaces:  web,  CLI,  API   ©  SysFera     SysFera-­‐DS  
27. DEMO   ©  SysFera     SysFera-­‐DS  
28. Next  version  of  the  WebBoard   ©  SysFera     SysFera-­‐DS   Responsive  thanks  to  Boostrap   (for  smartphones,  tablets,  PC)  
29. Summary   •  SysFera-­‐DS   – Web  portal:  WebBoard   – InteracIon  with  resources   •  Batch  scheduler   •  Vishnu   •  ROMEO  –  CRIHAN   – Access  to  remote  resources  directly  through  a   classical  web  browser   – Mutualize  compuIng  resources   – Batch  computaIon  or  visualizaIon   ©  SysFera     SysFera-­‐DS  
30. ©  SysFera     SysFera-­‐DS   www.sysfera.com   sysfera.github.com   www.sysfera.fr/webboard       contact@sysfera.com   Benjamin.Depardon@sysfera.com   Thanks!   Ques2ons?