SysFera-DS @ ROMEO-CRIHAN

SysFera-‐DS @ ROMEO – CRIHAN BUX, March
2013 Benjamin Depardon CTO@SysFera © SysFera HPC and SaaS made easy

SysFera •  2001-‐2010 – Laboratoire de l’InformaIque du parallélisme
(ENS, INRIA), Project-‐Team Graal/Avalon – Décrypthon Project with AFM, IBM •  2010-‐2013 – SysFera – Co-‐developing with EDF R&D – ParIcipate in compuIng plaZorm projects (IRT, ANR, FP7, Mésocentres, Décrypthon 3) © SysFera SysFera-‐DS

SysFera – Vision •  Ease of use of HPC
resources •  Researchers/engineers must concentrate on their ﬁeld of experIse, not on computer – Ease of use – Transparent access to resources – Federate access to resources © SysFera SysFera-‐DS

© SysFera SysFera-‐DS HTML5 Browser Mono-‐cluster
Mul2-‐cluster

ROMEO – CRIHAN © SysFera SysFera-‐DS
!!" LoadLeveler PBS Pro

ROMEO – CRIHAN © SysFera SysFera-‐DS
LoadLeveler PBS Pro

WEBBOARD © SysFera SysFera-‐DS

WebBoard -‐ PresentaIon •  CollaboraIve tool dedicated to HPC
–  Projects management –  Access rights management –  ApplicaIons management –  AccounIng, staIsIcs and acIvity –  Relies on an infrastructure manager (e.g., Vishnu, BS) •  Flexible: easily adaptable –  Remote visualizaIon plugin –  File management plugin –  Graphical UI is adaptable © SysFera SysFera-‐DS

WebBoard © SysFera SysFera-‐DS

WebBoard – ApplicaIons © SysFera SysFera-‐DS

WebBoard – GesIon de ﬁchiers © SysFera
SysFera-‐DS

WebBoard – Remote VisualizaIon © SysFera SysFera-‐DS

Remote visualizaIon © SysFera SysFera-‐DS

WebBoard -‐ Deployment •  Easy installaIon –  Standard
Tomcat and a war –  ConnecIon with main DB (Postgres, MySQL) –  Plugged to the infrastructure manager thanks to a message bus (RabbitMQ) •  ConfiguraIon with 1 file –  Host/Port/type for DB access –  Mail server configuraIon for noIficaIons –  Host/Port for RabbitMQ –  Plugins acIvaIon/deacIvaIon © SysFera SysFera-‐DS

WebBoard – Advantages •  Remote access to distant machines
through a simple web browser on the client side •  Mutualize computaIonal resources usage •  Batch computaIon or visualizaIon •  ReservaIon of remote visualizaIon sessions in advance •  Projects, access rights, applicaIons and ﬁles management © SysFera SysFera-‐DS

VISHNU MIDDLEWARE © SysFera SysFera-‐DS

Vishnu © SysFera SysFera-‐DS h@p://github.com/sysfera/vishnu

Vishnu – Components © SysFera SysFera-‐DS
Java API Python API Command line interface Vishnu API (C++) UMS client IMS client FMS client TMS client UMS server IMS server FMS server TMS server ZMQ Database SCP/RSYNC Batch schedulers Communication bus Vishnu interface software component Vishnu internal software component External components Client side Server side Dispatcher

Vishnu -‐ Deployment •  Distributed soluIon –  ConnecIon
with main DB (Postgres, MySQL) –  Daemons to deploy on each cluster’s gateway (TMS, IMS) –  “Central” daemons (UMS, FMS) –  OpIonally: dispatcher (proxy) •  ConfiguraIon with 1 file –  Host/Port/type for DB –  Mail server configuraIon for noIficaIons –  Host/Port for ZeroMQ –  Batch scheduler type •  Works in userspace © SysFera SysFera-‐DS

TMS – Generic script © SysFera Batch
scheduler speciﬁc parameters Batch scheduler generic parameters Generic variables Script’s parameters: strings or ﬁles SysFera-‐DS

Local accounts management © SysFera Site 1
/home/bobby Site 2 /home/bob1 Site 3 /home/bob SeD SysFera-DS End-User Computer SeD SeD Identiﬁcation DB (1) Open session (3) Send requests ... (6) Close session (4) Use global ID (5) ssh as Bobby (5) ssh as Bob1 (5) ssh as Bob (2) Retreive identification credentials LDAP1,2...n SysFera-‐DS

Vishnu – Advantages •  Mutualize compuIng resources • 
Non intrusive / non exclusive •  Remote access to distant resources through various API C++/Python/Java, and CLI •  Abstract layer to diﬀerent batch schedulers –  SLURM, SGE, Torque, PBS Pro, LoadLeveler, LSF –  Cloud: OpenStack (DeltaCloud) •  End-‐to-‐end users’ idenIty management •  Distributed and fault tolerant architecture © SysFera SysFera-‐DS

ROMEO-‐CRIHAN © SysFera SysFera-‐DS PBS Pro
LoadLeveler !"#!$%&'%$(!)&*% +",-$).),/01 '",2$3&4)$25%0$63&*% +",-$).),/01 ''%5603,&4)$25%0$63&*% SLURM '",2$3578 User1 HTML5 Browser User2 SSL Internet + Firewalls 9$34!"$:!;,).),/0 <,6=4#0 > ?!#! @,/03

SALOME & Vishnu •  Open-‐source mulI-‐physics plaZorm • 
Through its JOBMANAGER module, SALOME can: submit, follow, cancel jobs, transfer files before and aler jobs’ execuIon •  Benefits of using Vishnu – Access to a new machine can be done without any modificaIon in SALOME, whatever the BS – SALOME’s jobs can be managed through several interfaces: web, CLI, API © SysFera SysFera-‐DS

Summary •  SysFera-‐DS – Web portal: WebBoard – InteracIon
with resources •  Batch scheduler •  Vishnu •  ROMEO – CRIHAN – Access to remote resources directly through a classical web browser – Mutualize compuIng resources – Batch computaIon or visualizaIon © SysFera SysFera-‐DS

© SysFera SysFera-‐DS www.sysfera.com sysfera.github.com
www.sysfera.fr/webboard [email protected] [email protected] Thanks! Ques2ons?

SysFera-DS @ ROMEO-CRIHAN

SysFera-DS @ ROMEO-CRIHAN

SysFera

Transcript