The Origin of a New Species

You have heard about how functional programming is all about (the absence of) mutations. That is, you write your functions without modifying any states. Since your code does not directly modify any data, it is much easier for the language and runtime to figure out distribution of data and parallelizing your computations automagically for you. Coming from an imperative programming background as a Fortran/C/C++ programmer, you wonder if this is a total BS, if it is not, how you can also harness this power for your own scientific computing needs.

Your confusion will end with this talk. I will show you, using code for a particle-mesh simulation as an example, how to apply functional programming principles to scientific computing. You will learn how to use immutable data structures like Spark RDDs to generate and organize data and how to express computations using the usual purely functional operations and the higher-order, almighty, join operation. You will learn how the language and runtime help you, rather than work against you, on distributing data and parallelizing computations. You will see with your own eyes that it is no BS. You will believe in functional programming as well.

The absence of mutations gives rise to a new way of doing scientific computing. It is elegant and productive. It is parallelized and Cloud-ready the very first moment you start developing it. It is different, it is a New Species.

Transcript

1. The Origin of a New Species
2. A Collection-Query Based Approach for High Performance Scientific Code
3. A Collection-Query Based Approach for High Performance Scientific Code
4. A Collection-Query Based Approach for High Performance Scientific Code
5. Private Line 505-446-6374
6. PSA Pre
8. Angel Investor Science Advisor Chief Scientist Chief Technology Officier Entrepreneur
9. Physics
10. Physics
11. Technology Scala • Sweet spot between Object and Functonal programming • Higher order functions on collections: map, reduce, filter, flatMap • Easy transition from serial to distributed parallel programming • Second order functions on distributed collections: join, cogroup, cross
12. case class Position2(x: Length, y: Length) Position x y Particle id position case class Particle(id: Long,! position: Position2) Particle
13. Particle id position class ParticleSource {! def generate(id: Long): Particle! } class PointSource(position: Position2) extends! ParticleSource {! def generate(id: Long): Particle =! Particle(id, position)! } Particles id0 position id1 position Particle Source
14. Particles id position 0 1m, 2m 1 1m, 2m 2 1m, 2m 3 1m, 2m 4 1m, 2m 5 1m, 2m object ParticleGenerator {! def apply(particleSource: ParticleSource,! nParticles: Long)! (implicit sc: SparkContext): RDD[Particle] = {! val id = 0L until nParticles! ! sc.parallelize(id)! .map(particleSource.generate)! }! } Particle Generator
15. Wind case class Velocity2(u: Velocity, v: Velocity)! ! case class Cell(i: Long, j: Long)! ! object HomogeneousWindField {! def apply(xdims: Long, ydims: Long,! velocity: Velocity2)! (implicit sc: SparkContext):! RDD[(Cell, Velocity2)] = {! val ij = for {! i <- 0L until xdims! j <- 0L until ydims! } yield Cell(i, j)! ! sc.parallelize(ij).map(ij => (ij, velocity))! }! } Wind Field ij velocity 0, 0 1m/s, 1m/s 0, 1 1m/s, 1m/s 0, 2 1m/s, 1m/s 1, 0 1m/s, 1m/s 1, 1 1m/s, 1m/s 1, 2 1m/s, 1m/s
16. Turbulence Turbulence velocity -0.220m/s, 0.192m/s 0.007m/s, -0.486m/s -0.876m/s, -0.126m/s 0.119m/s, -0.460m/s -0.471m/s, 0.526m/s -0.003m/s, -0.526m/s object HomogeneousTurbulence {! def apply(sigma_u: Velocity, sigma_v: Velocity,! nParticles: Long)! (implicit sc: SparkContext): RDD[Velocity2] = {! val u = RandomRDDs! .normalRDD(sc, nParticles)! .map(x => x * sigma_u)! val v = RandomRDDs! .normalRDD(sc, nParticles)! .map(x => x * sigma_v)! ! u.zip(v).map { case (x, y) => Velocity2(x, y) }! }! }
17. Particle-Mesh Particles id position 0 1m, 2m 1 1m, 2m 2 1m, 2m 3 1m, 2m 4 1m, 2m 5 1m, 2m Particles in Mesh ij id position 1, 2 0 1m, 2m 1, 2 1 1m, 2m 1, 2 2 1m, 2m 1, 2 3 1m, 2m 1, 2 4 1m, 2m 1, 2 5 1m, 2m case class Distance2(x: Length, y: Length)! ! case class Position2(x: Length, y: Length) {! def -(position: Position2) =! Distance2(x - position.x, y - position.y)! }! ! val origin = Position2(0 meters, 0 meters)! val dxdy = Distance2(1 meters, 1 meters)! ! def toMesh(position: Position2): Cell = {! val distance = position - origin! val i = (distance.x / dxdy.x).toLong! val j = (distance.y / dxdy.y).toLong! Cell(i, j)! }! ! val particlesInMesh =! particles.map { particle =>! (toMesh(particle.position), particle)! }
18. Join Particles in Mesh ij id position 1, 2 0 1m, 2m 1, 2 1 1m, 2m 1, 2 2 1m, 2m 1, 2 3 1m, 2m 1, 2 4 1m, 2m 1, 2 5 1m, 2m Wind Field ij velocity 0, 0 1m/s, 1m/s 0, 1 1m/s, 1m/s 0, 2 1m/s, 1m/s 1, 0 1m/s, 1m/s 1, 1 1m/s, 1m/s 1, 2 1m/s, 1m/s Gone with the Wind ij id position velocity 1, 2 0 1m, 2m 1m/s, 1m/s 1, 2 1 1m, 2m 1m/s, 1m/s 1, 2 2 1m, 2m 1m/s, 1m/s 1, 2 3 1m, 2m 1m/s, 1m/s 1, 2 4 1m, 2m 1m/s, 1m/s 1, 2 5 1m, 2m 1m/s, 1m/s
19. Join val goneWithTheWind = wind.join(particlesInMesh)
20. Advection Gone with the Wind id position velocity 0 1m, 2m 1m/s, 1m/s 1 1m, 2m 1m/s, 1m/s 2 1m, 2m 1m/s, 1m/s 3 1m, 2m 1m/s, 1m/s 4 1m, 2m 1m/s, 1m/s 5 1m, 2m 1m/s, 1m/s Ashes id position 0 1.1m, 2.1m 1 1.1m, 2.1m 2 1.1m, 2.1m 3 1.1m, 2.1m 4 1.1m, 2.1m 5 1.1m, 2.1m case class Position2(x: Length, y: Length) {! def -(position: Position2) =! Distance2(x - position.x, y - position.y)! def +(distance: Distance2) =! Position2(x + distance.x, y + distance.y)! }! ! case class Velocity2(u: Velocity, v: Velocity) {! def *(time: Time) =! Distance2(u * time, v * time)! }! ! val ashes =! goneWithTheWind! .values! .map { case (v, p) =>! Particle(p.id, p.position + v * dt)! }
21. Perturbation Ashes id position 0 1.1m, 2.1m 1 1.1m, 2.1m 2 1.1m, 2.1m 3 1.1m, 2.1m 4 1.1m, 2.1m 5 1.1m, 2.1m Turbulence velocity -0.220m/s, 0.192m/s 0.007m/s, -0.486m/s -0.876m/s, -0.126m/s 0.119m/s, -0.460m/s -0.471m/s, 0.526m/s -0.003m/s, -0.526m/s Particles id position 0 1.078m, 2.119m 1 1.100m, 2.051m 2 1.012m, 2.087m 3 1.111m, 2.054m 4 1.052m, 2.047m 5 1.100m, 2.152m zip
22. Ashes to Ashes, Dust to Dust var particles = ParticleGenerator(PointSource(1 meters, 2 meters), 6)! val wind = HomogeneousWindField(3, 3, Velocity2(1 mps, 1 mps))! val turbulence = HomogeneousTurbulence(0.5 mps, 0.5 mps, 6)! ! val origin = Position2(0 meters, 0 meters)! val dxdy = Distance2(1 meters, 1 meters)! ! val dt = 0.1! for (t <- 0.0 to 5.0 by dt) {! val particlesInMesh =! particles.map { particle => (toMesh(particle.position), particle) }! ! val goneWithTheWind = wind.join(particlesInMesh)! ! val ashes = goneWithTheWind.values.map { case (velocity, particle) =>! Particle(particle.id, particle.position + velocity * (dt seconds))! }! ! particles = turbulence.zip(ashes).map { case (velocity, particle) =>! Particle(particle.id, particle.position + velocity * (dt seconds))! }! }
23. Cloud Deployment
24. Visualization
25. It’s All Rainbows and Unicorns
26. Houston, we have a problem
27. Scalability True More Physics More CS Other Physics Web Delivery