Asya Shklyar

ScaleConf Colombia 2018

Transcript

1. Building an HPC Environment at a Liberal Arts College Asya Shklyar April 2018
2. Who? • Director of HPC at Pomona since December 2017 • Building HPC-type environments since 2007 (454 Life Sciences/Roche) • In IT since 1995 (coaxial networks, b-tree databases, DOS 6.0, RH 4.0) • Consultant for 2 years focusing on cloud and data management and assessments • Built a couple of life sciences HPC infrastructures from scratch and revamped others • Bioinformatics Analyst • Java/perl/CGI/HTML/CSS, and recently Python, Go, Rust, Scala, Haskell • Systems Engineer https://www.pomona.edu/directory/people/asya-shklyar
3. Where? • Pomona College • Claremont, CA • 5c/7c • #1 on Forbes list in 2017 • Cal Poly, UC Irvine, UCLA, UCSD shadow https://www.pomona.edu/ http://www.claremont.edu/
4. Plan of action on the new job • Dec-Jan: Interview faculty and staff and research what others have done • Feb: Shop for hardware, choose a vendor • March: Order (April: wait!) *purchasing always takes 1-2 months • May: Build • 3-5 use cases over the summer (research programs) and train students • Integrate teaching in 2018 • Scale in 2019 • Build a local/distributed XSEDE-like grid by 2020 • The goal: offer a blueprint to other institutions
5. History of HPC at Liberal Arts Colleges • External link to another presentation • https://pomona.box.com/s/2jd2matpne2rcebxyec93pcwsqtz9dpc
6. Challenges • Limited budget • Lack of institutional familiarity with HPC (and associated concepts) • Software not parallelized or version-controlled in most cases • Remote visualization does not exist; no GPUs • Data silos (local disk, permissions all over the place) • Authentication not using AD • Not benchmarked/profiled; no idea how long anything is supposed to take • Research and teaching combined • External collaborators, even international (GDPR) • Politics (5c/7c) • Pomona IT “seeding” the environment
7. Nice-to-Haves or Must-Haves? • One of everything – because research - and the software will vary • As high as possible core count but also a high Ghz server • GPUs to augment CPU cores; NVLINK’ed for DL • Fast shared storage for when you don’t want to shuffle data or local scratch is not enough • Central authentication • Versioning • Configuration Control • Dockerized-everything for portability; virtualization too slow
8. Hardware Choices • Intel, established and pretty standard • AMD, cheaper? More cores in the latest Epyc • x86 for software compatibility • GPUs for CPU augmentation • Cray, PowerPC, Itanium not considered because of the budget/complexity • Same goes for ARM, FPGA, ASICs
9. Intel vs AMD Cores vs Ghz Sockets - Latest Intel Skylake (28) - Latest AMD Epyc (32) - Previous Intel Broadwell (22) - No previous AMD gen to speak of - 3.6 Ghz highest – less cores - 2 socket (vs 4 or 8) - 12/24/32 DIMMs=channels (split between CPUs) - 16/32/64/128 GB chips - Max 3 TB; reasonable 256-512-768 GB https://www.nextplatform.com/2017/07/11/x86-battle-lines-drawn-intels-skylake-launch/
10. NVIDIA GPU vs ? • AMD Instinct (Vega) server card announced but not qualified by vendors • Tesla – NVIDIA server card • V100 – latest Tesla – Volta architecture • P100 – previous Tesla – Pascal architecture • Other server GPUs (virtualization/CAD etc) • Desktop GPUs not considered although an external eGPU (1080Ti) is used for proof of concepts
11. NVIDIA-specific support - With the purchase of a DGX, DGX-1 or DGX-2 only - Access to NVIDIA Cloud (Dockerized applications - free) - Can be run locally on your own infrastructure or AWS - DIGITS ( https://developer.nvidia.com/digits ) - Online and in-person training resources - Access to User Group - Pre-installed/calibrated NVLINK (4, 8, 16) - Microway and some other vendors offer SuperMicro-based equivalent, cheaper but unsupported by NVIDIA* * Also testing!
12. 3 vendors considered and finalized • Custom SuperMicro vs custom non-SuperMicro • NVIDIA rolls its own, no avoiding that but the chosen vendor is a reseller (NVIDIA certifies resellers) • NVLINK support either way • Cheapest chosen, and majority of equipment from same vendor to simplify purchasing and support • 1 year vs 3 year support consideration • OS is free (but would not be with Red Hat) • CMU also from the same vendor
13. Cluster Provisioning and Management • Bright Computing • CMU/HP does not compare • XCAT is discontinued • Rocks is still kicking but not reliable • Bright supports Bare Metal, Virtualization or Docker or all • Hardware tested, drivers available as well as support • OpenStack/Hadoop/Spark stacks and ML/DL stacks included • Cloud bursting support • Simple NFS-mounted software provisioning or Image-based for performance http://www.brightcomputing.com/ • PXE boot • Image Management • Software Management • DNS and DHCP • NTP • AD/LDAP Integration, sssd caching/updates • Metrics/Health • Concept of Head Node/Login Node • HA
14. Operating Systems • CentOS 7.4 • Latest stable • Most software works on CentOS/Red Hat/Scientific Linux/Fedora • Ubuntu inevitable (NVIDIA DGX) • Older OS support (Legacy Software); DOCKERIZE! • Windows if necessary
15. Post-Bare-Metal Provisioning • Ansible (vs Puppet or Chef or SaltStack) • Agentless • Well-supported (Red Hat) • Visualization Tools (Ansible Tower, Rundeck) • Easy to pick up
16. Git/GitHub for everything • Subversion in existence but not suitable, scalabilty, speed • README.md for every section • Structure of the repo • System-specific files along with documentation
17. Scheduler • Slurm • Accounting • XDMOD • Fair Share • Queues for short/long, single core/multi-core, prod/dev, interactive, RAM- intensive • Pre-job and post-job (moving data) • Batch scripts with arguments, input, output • Time Limit is important • Chargeback (Grants!) https://slurm.schedmd.com/ http://open.xdmod.org/
18. Networking • Mellanox Ethernet • SN2700B • 32 port • $$ • 10/40 GB • 100 GB considerably more expensive • Infiniband overkill (MPI) • 2 x one for compute and one for storage, with VLANs for failover • QSFP+ • Dual port cards in nodes
19. Storage • Existing Windows Home Directories • Centralized auth with AD • Pure Storage for parallel processing, All-Flash, NVMe • 512/1/2 TB • 20 slots • M10, M20, etc • Local scratch • Intel Optane • Future: BeeGFS (small/large/mixed) • Future: Object Storage for cold/deep • Future: Data Lake https://www.purestorage.com/
20. Monitoring • Bright Built-in – best for real time although supports historical • Slurm accounting+XDMOD for “how long do the jobs run” and “how many”, and “what type”, and “who is hogging the queue” • Hosted ELK for logs and anomaly detection • check_mk for pulse/alarms • Nessus Pro/Tenable – vulnerability/patching • Future: Riemann for correlation
21. Support • An Applications Manager who knows everyone! • Lots of personal communications required • Attending faculty meetings • 1-2 students, summer interns • Plans to grow the team as the environment stabilizes with the focus on software support
22. Software • COMSOL • Gaussian • MatLab • R • Mathematica • SPSS • Stata • Tableau • fNIRS/fMRI processing • Deep Learning/Inference • Science Gateways • Jupyter/Hub/Lab … hundreds of items, potentially …license management … user customization (venv etc)
23. Kubernetes for scaling established workflows http://www.nvidia.com/object/docker-container.html https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ https://www.docker.com/kubernetes
24. minio (or weka.io) for container persistent shared storage https://www.minio.io/docker.html
25. Exploring • VR/AR/MR/XR • Blockchain use (credentials etc)
26. Questions-Comments? • Asya.Shklyar@pomona.edu