Configuring a Windows Azure - Hybrid HPC cluster

A 5 minute review of the Microsoft Windows HPC platform and how to augment an existing on premise installation with Azure worker nodes to create a Hybrid HPC - Azure configuration.

Transcript

1. Chris Condo ccondo@outlook.com August 20, 2014 Azure & HPC Configuration
2. HPC = High Performance Computing • HPC is Microsoft’s “Big Compute” platform • Parallel and batch compute capabilities. • Applications: • Actuarial sciences. • Financial Risk Modeling • Digital content creation, transcoding. • Genetics research.
3. • Head Node hosts a control panel which allows you to create cluster configurations. • The head node sends work the compute node clusters. • The work is split up into work units and executed in parallel. Typical on premise HPC configuration HEAD NODE ON PREMISE COMPUTE NODES
4. Hybrid HPC-Azure configuration HEAD NODE ON PREM COMPUTE NODES AZURE COMPUTE NODES • Add capacity with worker nodes in the cloud. • Useful for scenarios such as seasonal bursts where extra capacity is needed occasionally. • Its not necessarily about speed, its about throughput.
5. Azure Compute Node Size • Use either A8 or A9 class machines. • Problem sets can be configured to be processed in parallel across 8 or 16 cores. • These machines come at an added cost. • Managing that resource is important to controlling $$$ spent.
6. Cluster Manager running on the head node
7. Cluster Manager starting Azure Nodes. • There’s manual steps needed to start/stop jobs. • These steps can be automated via PowerShell
8. Powershell can replace manual interaction. • Your application can be written to execute Powershell to add Azure capacity as needed. • Add nodes to your HPC Cluster dynamically • Get-HpcNodeTemplate $HpcNodeTemplate -Scheduler $Scheduler | Add- HpcNodeSet -Quantity $Quantity -Size $Size -Scheduler $Scheduler • Start or Stop the nodes • Get-HpcNodeTemplate $HpcNodeTemplate -Scheduler $Scheduler | Start- HpcNodeSet -Scheduler $Scheduler • Shut down the nodes when the job is done, don’t waste $$$ with a VM that’s not doing work.
9. Windows Azure Management Libraries • A Natural .NET Framework that maps cleanly to the underlying REST API • Introduced late 2013 • Supports the Portable Class Library (PCL). • Ships as a NuGet Package you install from VS • Designed to use C# 5 async tasks • Easy to code with C# using Microsoft.WindowsAzure; using Microsoft.WindowsAzure.Management.Compute; using Microsoft.WindowsAzure.Management.Storage;
10. Do cool stuff like create Azure VMs from a desktop. • Deploy cloud services from your .Net applications • Eg: Create a VM internal async Task CreateVM() { await _computeManagementClient.VirtualMachines.CreateAsync( “MyAzureService", “MyAzureDeployment", new VirtualMachineCreateParameters { RoleName = “MyName", RoleSize = VirtualMachineRoleSize.ExtraSmall }, new System.Threading.CancellationToken()); }
11. Summary • HPC is specialized for certain industries that require massive computing power and can make use of parallel processing. • Azure “Burst to cloud” is an effective means to add capacity. • Automating the starting and stopping of your Azure resources can help control costs. • Powershell and/or WAML are 2 ways to develop operational scripts to automate Hybrid-HPC work management.