@RayHightower All Things Open October 13-15, 2019 Raleigh, NC USA Raspberry Pi is a Gateway Drug for GPUs

What Why How

Parallelism: Boosting performance by executing at least two threads at the same time.

RayHightower.com

No content

No content

Why parallelism? Because Moore’s Law is running out.

Transistors on a wafer of silicon.

Is this text too small for us to read?

Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read?

Is this text too small for us to read? Is text too small for us to read? Is this tex too small for us to read? Is this text too small for us to read? Is this text too sm for us to read? Is this text too small for to read? Is this text too small for us to read? Is this text too small for us to rea Is this text too small for us to read? Is text too small for us to read? Is this tex

ext too small for us to read? Is this text too small for us to read? Is this text to mall for us to read? Is this text too small for us to read? Is this text too small s to read? Is this text too small for us to read? Is this text too small for us to ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm

to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is

to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is Limited by Eyesight

Limited by Physics to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is

Parallelism

Concurrency At least two threads are making progress. Parallelism At least two threads are executing simultaneously. Oracle Multithreaded Programming Guide http://docs.oracle.com/cd/E19455-01/806-5257/6je9h032b/index.html vs.

Parallelism Keyboard Mouse Disk Time ======> Concurrency Task Switching K M D K M D K M D K M D K M D K M D Time ======>

How?

Embarrassingly parallel problem.

Embarrassingly parallel problem. … like finding prime numbers.

No content

Model 3 Model 2 Model 1

No content

No content

ssh-keygen

authorized_keys

RPi01 RPi02 RPi03 RPi04 authorized_keys

No content

The cluster works!

No content

No content

RPi Summary •Assemble the cluster. •Communicate via mpi4py. •Assign a portion of the problem to each node. •Gather the results.

What’s next?

Quick detour… Parallella

If one ox could not do the job they did not try to grow a bigger ox, but used two oxen. When we need greater computer power, the answer is not to get a bigger computer, but to build systems of computers and operate them in parallel. -Grace Hopper

RJ-45 Power μUSB μHDMI μSD

No content

No content

Summary: Finding Primes 0" 50" 100" 150" 200" 250" Parallel Parallella Serial Mac Serial Parallella 18.6 sec 14.4 sec 237.1 sec ($2,000.00 Apple MacBook Pro) ($150.00 Parallella)

What’s next?

Graphical Processing Unit

MacBook Pro Video 384 cores

Raspberry Pi is a gateway for the GPU.

No content

No content

No content

No content

No content

No content

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO Adapteva

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO Adapteva It’s hard.

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO Adapteva It’s hard. And it’s worth it.

Moore’s Law: 2x every 18 months

Moore’s Law for GPUs https://www.nextbigfuture.com/2017/04/for-commoners-using-cpus-meaningful.html Moore Nvidia

# of Nodes Total Render Time Power Cost (5yrs) System Cost CPU - Dual Skylake 25 38 hours $70k $250k Nvidia RTX Server 1 6 hours $10k $30k “Pixar artists already rely on Nvidia ray tracing, and RTX more than doubles the performance they will see. We’re excited to use RTX on our upcoming films.” -Steve May, CTO, Pixar From Jensen Huang’s keynote at Nvidia GTC 2019

Nvidia Jetson Nano 128 GPU cores 4 CPU cores

No content

Compute Unified Device Architecture

CUDA supports Python, Clojure, C, C++, C#, Fortran, Ruby, Java…

No content

Embarrassingly parallel problem. … deep learning.

No content

No content

•Training •Testing •Deployment

Avoiding “gotchas” •Lots of data/examples •Labeling. •Randomly choose 20% for testing. •Don’t train w/testing data.

Parallelism accelerates deep learning.

No content

No content

No content

Resources

No content

https://www.youtube.com/watch?v=aircAruvnKk&list=PLOnWKC1gI_OOIrkRZDq2SgYoDxfRAJtdF&index=66&t=0s Recommended by the Nvidia instructors.

http://alexlenail.me/NN-SVG/index.html

No content

No content

No content

No content

Summary… •Parallelism + RPi •Jetson Nano •Deep Learning •Resources

In business, there is nothing more valuable than a technical advantage your competitors don’t understand. -Paul Graham http://www.paulgraham.com/avg.html

Thanks! RayHightower.com