Slide 1

Slide 1 text

The Fast and The Dangerous: Safer Parallel Programming with Types Nicholas Ng Students Seminar 24 Jan 2014 Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 2

Slide 2 text

Me Nick 4th year PhD student Mobility Research Group and Custom Computing Group Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 3

Slide 3 text

Parallel Programming Multicore processors, computer clusters Improve software performance by parallelising Writing software to make use of the resources in parallel A single application to do multiple tasks at the same time Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 4

Slide 4 text

Parallel Programming Shared-memory parallel programming Parallelism on the same machine (e.g. Multicore) Distributed parallel programming Parallelism by using many machines (e.g. Computer clusters) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 5

Slide 5 text

Parallel Programming Shared-memory parallel programming Parallelism on the same machine (e.g. Multicore) Threads, critical sections, locks Implicit (e.g. MATLAB?) Distributed parallel programming Parallelism by using many machines (e.g. Computer clusters) Message-passing i.e. Coordinate by sending messages between processes Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 6

Slide 6 text

Message-Passing Interface MPI: a 20 years old standard Distributed parallel programming w/ message-passing Widely used in scientific applications C and FORTRAN binding Scientists: not just software engineers, CS researchers Single Program, Multiple Data (SPMD) One source code, executes multiple processes Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 7

Slide 7 text

Message-Passing Interface 1 #include 2 int main(int argc, char *argv[]) 3 { 4 5 6 7 8 9 10 11 12 return EXIT_SUCCESS; 13 } Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 8

Slide 8 text

Message-Passing Interface 1 #include 2 int main(int argc, char *argv[]) 3 { 4 MPI_Init(&argc, &argv); 5 // 6 // 7 // Body of MPI Program 8 // 9 // 10 // 11 MPI_Finalize(); 12 return EXIT_SUCCESS; 13 } Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 9

Slide 9 text

Message-Passing Interface 1 #include 2 int main(int argc, char *argv[]) 3 { int rank, size; 4 MPI_Init(&argc, &argv); 5 MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Process ID 6 MPI_Comm_size(MPI_COMM_WORLD, &size); // Total procs 7 // 8 // Calculation goes here 9 // 10 // 11 MPI_Finalize(); 12 return EXIT_SUCCESS; 13 } Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 10

Slide 10 text

Message-Passing Interface 1 #include 2 int main(int argc, char *argv[]) 3 { int rank, size, buf[20]; 4 MPI_Init(&argc, &argv); 5 MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Process ID 6 MPI_Comm_size(MPI_COMM_WORLD, &size); // Total procs 7 8 if (rank == 1 ) MPI_Recv(buf, 20, MPI_INT, 0 , ...); 9 if (rank == 0 ) MPI_Send(buf, 20, MPI_INT, 1 , ...); 10 11 MPI_Finalize(); 12 return EXIT_SUCCESS; 13 } Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 11

Slide 11 text

Message-Passing Interface Process 0 1 MPI_Init(); 2 MPI_Comm_rank(rank= 0 ); 3 MPI_Comm_size(size=2); 4 5 if (rank == 1) 6 MPI_Recv(0); 7 if (rank == 0 ) 8 MPI_Send( 1 ); 9 10 MPI_Finalize(); Process 1 1 MPI_Init(); 2 MPI_Comm_rank(rank= 1 ); 3 MPI_Comm_size(size=2); 4 5 if (rank == 1 ) 6 MPI_Recv( 0 ); 7 if (rank == 0) 8 MPI_Send(1); 9 10 MPI_Finalize(); Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 12

Slide 12 text

Message-Passing Interface Process 0 1 MPI_Init(); 2 MPI_Comm_rank(rank= 0 ); 3 MPI_Comm_size(size=2); 4 5 if (rank == 1) 6 MPI_Recv(0); 7 if (rank == 0 ) 8 MPI_Send( 1 ); 9 10 MPI_Finalize(); Process 1 1 MPI_Init(); 2 MPI_Comm_rank(rank= 1 ); 3 MPI_Comm_size(size=2); 4 5 if (rank == 1 ) 6 MPI_Recv( 0 ); 7 if (rank == 0) 8 MPI_Send(1); 9 10 MPI_Finalize(); Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 13

Slide 13 text

The message passing paradigm Easily scalable e.g. just use MPI_Send(rank+1), spawn 1000 processes A bit tedious :( The sender has to know who the receiver is The receiver has to know who the sender is Communication mismatch: most common error Message passing paradigm in general (not just MPI) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 14

Slide 14 text

The message passing paradigm Easily scalable e.g. just use MPI_Send(rank+1), spawn 1000 processes A bit tedious :( The sender has to know who the receiver is The receiver has to know who the sender is Communication mismatch: most common error Message passing paradigm in general (not just MPI) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 15

Slide 15 text

Meanwhile.. Cilk (MIT/Intel) Unified Parallel C (UPC) - Multiple implementations Async-await workflow - F# Functional languages* - Haskell/ghc Partitioned Global Address Space (PGAS) - X10, Chapel Are these new languages/models being adopted? Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 16

Slide 16 text

Debugging Hard to fix by inventing new language/models Make errors easier to find (Commercial) debuggers that understand parallel context Running multiple debuggers in parallel per thread/process Runtime solutions: sometimes error will happen, sometimes not Can we check these errors before the program is executed? Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 17

Slide 17 text

Type systems Type systems? Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 18

Slide 18 text

Type systems Been around as long as programming languages Types refer to data types (also called ”sorts”) Probably use it every day without knowing Extremely important for functional programming languages e.g. Haskell, ML Abstraction for the underlying functionalities Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 19

Slide 19 text

Types 1 int number = 52; // OK 2 int number = add(1, 13); // OK 3 int number = "Nick"; // Error! Expression/statement Type variable number int (integer) value 52 int function call add(1, 13) function returning int value "Nick" String (list of characters) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 20

Slide 20 text

Types (2) Type errors are picked up by compilers Known as static type checking Does not interfere with execution Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 21

Slide 21 text

Types (2) Type errors are picked up by compilers Known as static type checking Does not interfere with execution But what does it have to do with parallel programming? Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 22

Slide 22 text

Session Types Traditional types Data types: int ←→ int Makes sure the variable and the value are compatible Useful to abstract and proof properties of a program Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 23

Slide 23 text

Session Types Traditional types Data types: int ←→ int Makes sure the variable and the value are compatible Useful to abstract and proof properties of a program Session Types: a different kind of Types Communication always involves two sides Session types: send ←→ receive Types across two processes, i.e. types for communication Makes sure the communication is compatible Useful to abstract and proof properties of a communication Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 24

Slide 24 text

Session Types (2) Session Type refers to the ”type” of the whole program A series of send/receive with control-flow structure Also called session or protocol Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 25

Slide 25 text

Scribble/Pabble language 1 global protocol OneShot(role P[0..10]) { 2 3 4 Left(int) from P[0] to P[1]; 5 6 7 8 9 } Developer friendly ”session types” Sending/receiving message Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 26

Slide 26 text

Scribble/Pabble language 1 global protocol OneShot(role P[0..10]) { 2 3 choice at P[0] { 4 Left(int) from P[0] to P[1]; 5 } or { 6 Right(int) from P[0] to P[1]; } 7 8 9 } Developer friendly ”session types” Sending/receiving message Branching (conditionals) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 27

Slide 27 text

Scribble/Pabble language 1 global protocol OneShot(role P[0..10]) { 2 rec X { 3 choice at P[0] { 4 Left(int) from P[0] to P[1]; 5 } or { 6 Right(int) from P[0] to P[1]; } 7 continue X; 8 } 9 } Developer friendly ”session types” Sending/receiving message Branching (conditionals) Recursion (loops) Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 28

Slide 28 text

My work Applying Session Types on parallel programming Static type checking Match program with ”expected” type Type-checkable ⇒ program 100% has no communication mismatch Errors detected before even running! Generating code from session types Correct communication code by design Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 29

Slide 29 text

Session Type checking on MPI Type 1 local protocol MPI at P( 2 role P[0..1]) { 3 4 5 if P[ 1 ] 6 int from P[ 0 ]; 7 if P[ 0 ] 8 int to P[ 1 ]; 9 } MPI program 1 MPI_Init(); 2 MPI_Comm_rank(rank= 0 ); 3 MPI_Comm_size(size=2); 4 5 if (rank == 1 ) 6 MPI_Recv( 0 ); 7 if (rank == 0 ) 8 MPI_Send( 1 ); 9 MPI_Finalize(); Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 30

Slide 30 text

Challenges Type checking is hard.. What if sender receiver are not known at compile time? Language feature abuse (e.g. most use of pointers) Open-ended expressions (how to match with type) Can we type-check complex programs automatically Without manual annotations hints for checker Without changing source code Removing/changing code that confuses checker Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 31

Slide 31 text

Challenges (2) High performance computing community Better performance Optimise at execution time Runtime adaptation as needed Formal systems community Stronger safety assurance Get program 100% correct at compile time Everything should execute as planned Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 32

Slide 32 text

Conclusion Parallel programming is difficult Communication mismatch Most common error for message-passing parallelism Session Types: a typing system for communication Parallel programming with session types Spotting communication error early Using types: No penalty on performance Questions? Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 33

Slide 33 text

Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 34

Slide 34 text

From Global to Local Types Write this 1 global protocol Main 2 (role P[0..1]) { 3 4 int from P[0] to P[1]; 5 6 } Generate this 1 local protocol Main at P 2 (role P[0..1]) { 3 4 if P[1] int from P[0]; 5 if P[0] int to P[1]; 6 } Always design in global view A higher-order abstraction Every pair of communication are compatible Nicholas Ng ([email protected]) Safer Parallel Programming with Types

Slide 35

Slide 35 text

From Global to Local Types Write this 1 global protocol Main 2 (role P[0..1]) { 3 4 int from P[0] to P[1]; 5 6 } Generate this 1 local protocol Main at P 2 (role P[0..1]) { 3 4 if P[1] int from P[0]; 5 if P[0] int to P[1]; 6 } Always design in global view A higher-order abstraction Every pair of communication are compatible Nicholas Ng ([email protected]) Safer Parallel Programming with Types