Discussing the What, Why and How to of NeSI

A discussion of what the New Zealand eScience Infrastructure (NeSI) is, what it offers and how researchers can benefit from its HPC, storage and support services.

Transcript

1. What, Why and How to of the New Zealand eScience Infrastructure AUT University, 7 August 2012 Ben Roberts & Tim McNamara
2. Overview • What does HPC enable? • What is NeSI? • People • HPC (CPU & GPU) • Data • Why NeSI was established? • How to access services • Access Process • Pricing • How to use the services • Job submission • Upload and share data
3. What science do computers enable? In particular, what do high performance computers (HPC) enable?
4. Note: a very incomplete view
5. observation theory
6. observation simulation theory
7. observation simulation theory HPC HPC Data processing & analysis Finer resolution & more variables
8. What is NeSI?
9. Technical
10. computation storage authentication
11. Systemic
12. people technology organisations
13. What is NeSI focused on?
14. Creating an advanced, scalable computing infrastructure to support New Zealand's research communities. International scale as opposed to institutional or departmental scale. Providing the necessary conditions for best possible uptake of this infrastructure, including grid middleware, research tools and applications, data management, user-support and community engagement. Encouraging a high level of co-ordination and co-operation within the research sector. We aim supplement instutitional resources to enable excellent science everywhere. Contributing to high quality research outputs by applying advanced computing and data management techniques and associated services.
15. What is an HPC facility?
16. • Batch processing • Use UNIX derivative operating system • Distributed file system hides network complexity • You ask for things to be installed or compiled => Your research goes much faster, but the process is different From a researcher's point of view
17. From a technical perspective • Capability • Maximum computing power available • Specialist hardware • Capacity • Cost-effectiveness focus • Commodity hardware • Some differences • Interconnect • Processor performance • Reliability • Sustained performance (e.g. % of time at peak)
18. What HPC facilities does NeSI have?
19. POWER6 x86 BlueGene/P POWER7 x86
20. POWER6 x86 BlueGene/P POWER7 x86 8198 0.8 GHz cores (2048 quad core CPUs split into partitions of 256) 1856 4.7 GHz cores (58 nodes, each with 32 CPUs) 416 3.3 GHz cores (13 nodes, each with four 8 core CPUs) 912 2.67 GHz cores (76 nodes, each with two 6 core CPUs) 40 3.03 GHz cores (5 nodes, each with dual 4 core CPUs)
21. POWER6 x86 BlueGene/P POWER7 x86 8198 0.8 GHz cores (2048 quad core CPUs split into partitions of 256) 1856 4.7 GHz cores (58 nodes, each with 32 CPUs) 416 3.3 GHz cores (13 nodes, each with four 8 core CPUs) 912 2.67 GHz cores (76 nodes, each with two 6 core CPUs) 40 3.03 GHz cores (5 nodes, each with dual 4 core CPUs) 8192 GB RAM (1GB per core) 5376 GB RAM (some 64GB nodes, others 128 GB) 1664 GB RAM (128 GB RAM per node) 7296 GB RAM (96 GB RAM per node) 280 GB RAM (96 GB RAM per node)
22. POWER6 x86 BlueGene/P POWER7 x86 8198 0.8 GHz cores (2048 quad core CPUs split into partitions of 256) 1856 4.7 GHz cores (58 nodes, each with 32 CPUs) 416 3.3 GHz cores (13 nodes, each with four 8 core CPUs) 912 2.67 GHz cores (76 nodes, each with two 6 core CPUs) 40 3.03 GHz cores (5 nodes, each with dual 4 core CPUs) 8192 GB RAM (1GB per core) 5376 GB RAM (some 64GB nodes, others 128 GB) 1664 GB RAM (128 GB RAM per node) 7296 GB RAM (96 GB RAM per node) 4 nodes with 2 Tesla M2090 GPUs 280 GB RAM (96 GB RAM per node) 5 nodes 2 Tesla M2070Q GPE GPUs
23. POWER6 x86 BlueGene/P POWER7 x86 8198 0.8 GHz cores (2048 quad core CPUs) 1856 4.7 GHz cores (58 nodes, each with 32 CPUs) (50 new nodes on the way) 416 3.3 GHz cores (13 nodes, each with four 8 core CPUs) 912 2.67 GHz cores (76 nodes, each with two 6 core CPUs) 40 3.03 GHz cores (5 nodes, each with dual 4 core CPUs) 8192 GB RAM (1GB per core) 5376 GB RAM (some 64GB nodes, others 128 GB) 1664 GB RAM (128 GB RAM per node) 7296 GB RAM (96 GB RAM per node) 4 nodes with 2 Tesla M2090 GPUs 280 GB RAM (96 GB RAM per node) 5 nodes 2 Tesla M2070Q GPE GPUs (~50 new nodes on the way)
24. Centre of eResearch
25. Applications Preinstalled (some licensing may be required): • Math: Gap, Magma, Matlab, Mathematica, R • BioInformatics: BLAST, BEAST, beagle, PhyML, MrBayes, BEDtools, Bamtools, Bowtie, Clustal Omega, Cufflinks, FastQC, FASTX Toolkit • Computational Chemistry: Gaussian, Gromacs, AMBER, Orca, VASP • Engineering: Ansys, Abaqus, OpenFOAM • Meteorology: WRF, WPS Scientific libraries Compilers: Fortran, C & C++ (gcc, Intel & PGI) BLAS, LAPACK/LAPACK++, ATLAS, FFTW, … Support for custom built applications: Batch submission (non-interactive, non-GUI, preferably parallel processing) Compilers (GNU, PGI C/C++ and Fortran, Intel, Java, Python, OpenMPI) CeR NeSI Staff: Service Delivery Manager: Marcus Gustafsson Systems Engineers: Yuriy Halytskyy, Aaron Hicks, + 1 TBD HPC Specialist Programmers: Markus Binsteiner, Martin Feller, Ben Roberts, Gene Soudlenkov, + 2 TBD
26. NIWA HPCF
27. Operating Environment SSH/X logon (via dedicated NeSI login node) LoadLeveler (Batch Queue) to run non-interactive jobs IBM (xl) Fortran (77, 90, 95, 2003), C and C++ compilers IBM High Performance Computing Toolkit (MPI, OpenMP, etc;) TotalView graphical debugger Third Party Software (e.g.): • Make, Cmake, Python, Java, git, Subversion, GSL, Hypre, LAPACK, ParMETIS, FFTW, NetCDF (3 & 4), parallel-NetCDF, HDF5, jasper, VisIt; • Access any set of specific s/w versions via MODULES NIWA NeSI Support Staff • Service Delivery Manager: Michael Uddstrom (0.05 FTE) • Systems Engineers: Chris Edsall, Fabrice Cantos (0.21 FTE each) • HPC Specialist Scientific Programmer: Mark Cheeseman (0.21 FTE) • Expected system uptime: >99.5%
28. Unified Model (UK Met Office / Hadley Centre) Weather forecasting: (global & regional – to 100 m resolution) 3DVAR & 4DVAR data assimilation Regional Climate modelling HadGEM3-RA Coupled (atmospheric, ocean, land, sea ice) earth simulation HadGEM3 Chemistry Climate Modelling – UKCA Ocean Modelling ROMS (Regional Ocean Model) NEMO. (Global Ocean Model) Wave Modelling WaveWatch 3, SWAN CFD Gerris (self refining grid) Typical job sizes: 64 – 1024 cores & O(10) GB output per job
29. BlueFern
30. IBM iDataPlex Visualisation Cluster 5 nodes (8 cores / SMP node) • 40 3.03 GHz Intel Xeon cores • 2 Tesla M2070Q GPUs / node • 96 GB Memory / node Applications • BG/P • Molecular Dynamics (NAMD, LAMMPS, VASP etc), Weather Forecasting (WRF), Protein Folding/Docking (AMBER, GROMACS, etc), Monte Carlo & researcher codes • P755/POWER7 • Fluid Dynamics (Fluent/CFX), Genomics (MrBayes etc), Numerical (Octave, R), interpreted languages (Java, Python+SciPy/NumPy) & researcher codes • Visualization • Visualization tools (VTK, ParaView, VisIt etc.) and high-speed remote graphical sessions (vizstack, turboVNC, etc.)
31. How do we get access?
32. Via an Allocation Class.
33. What's an Allocation Class?
34. Private Industry Research Teaching Proposal Development Collaborator
35. Private Industry Research Teaching Proposal Development Collaborator Three of most interest to this audience
36. Proposal Development Allocation Class • Price: Free • Support: 100% • Timeframe: Short-term • Question Answered: Is my research suited to NeSI? • Description: Can be considered a trial period. Acts as an investigation period to see whether research projects fit the NeSI model. Suited for cases where there may need to be a high level of support provided to researchers.
37. Research Allocation Class • Price: 20% of price* • Support: 100% • Timeframe: Long-term • Questions Answered: All research goals. • Description: NeSI HPC facilities for research projects. Higher requirements, but far larger allocations. * If you have no funds specified for HPC in your research budget, still talk to us. We will provide you access for free, with a reduced priority.
38. Teaching Allocation Class • Price: tbc* • Support: 100% • Timeframe: tbc • Questions Answered: Providing students confidence on HPC facilities. • Description: Add HPC to your curriculum, by utilising NeSI's HPC facilities within a classroom setting. • Caveat: NeSI is still working through the details with the research sector. Please talk to us if you might be interested in exploring this option.
39. How do I write a good application?
40. What qualifies as good science?
41. How scalable do my codes need to be?
42. What is the project lifecycle?
43. computation apply technical review seek funding publicity
44. Why should I bother?
45. • Bigger and faster machines • More likely to be able to be able to get many cores • Much more memory available • Dedicated staff • Installation provided • Compilation provided • Domain-specific expertise
46. Which domains is NeSI interested in?
47. All.
48. Seriously.
49. NeSI is open to access proposals from all disciplines at all institutions.
50. When should I talk to NeSI?
51. • Early is best • Encounter a barrier on a workstation • Can't access institutional resources • When you want to try a new approach • When you need something to run faster • Writing a grant proposal
52. You mentioned data storage earlier?
53. Data Fabric Federated data storage to support research projects. Primary use case (currently) is to facilitate rapid transfer between institutions. Hosted at The University of Auckland and Univeristy of Caterbury. Implemented with iRODS (Integrated Rule Orientated Data Service) Uses Tuakiri for authentication. Interfaced via HTTP, WebDAV, FUSE & GridFTP.
54. How do I talk to NeSI?
55. NeSI Central: http://www.nesi.org.nz/ Access Policy: http://www.nesi.org.nz/access-policy Eligibility: http://www.nesi.org.nz/eligibility Allocation Classes: http://www.nesi.org.nz/allocations Application Forms: http://www.nesi.org.nz/apply Calls Timetable: http://www.nesi.org.nz/timetable Storage: http://www.nesi.org.nz/files-and-data Case Studies: http://www.nesi.org.nz/case-studies Sites: CeR: http://www.eresearch.auckland.ac.nz/uoa/ NIWA: http://www.niwa.co.nz/our-services/hpcf UC: http://www.bluefern.canterbury.ac.nz/ NeSI staff are both here to help, and willing to help!