@RayHightower Nebraska.Code() August 14-16, 2019 Lincoln, NE USA Exploring Parallelism w/Raspberry Pi & Jetson Nano

Model 3 Model 2 Model 1

What Why How

Parallelism: Boosting performance by executing at least two threads at the same time.

RayHightower.com

No content

No content

Why parallelism? Because Moore’s Law is running out.

Transistors on a wafer of silicon.

Is this text too small for us to read?

Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read?

Is this text too small for us to read? Is text too small for us to read? Is this tex too small for us to read? Is this text too small for us to read? Is this text too sm for us to read? Is this text too small for to read? Is this text too small for us to read? Is this text too small for us to rea Is this text too small for us to read? Is text too small for us to read? Is this tex

ext too small for us to read? Is this text too small for us to read? Is this text to mall for us to read? Is this text too small for us to read? Is this text too small s to read? Is this text too small for us to read? Is this text too small for us to ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm or us to read? Is this text too small for us to read? Is this text too small for us ead? Is this text too small for us to read? Is this text too small for us to read? his text too small for us to read? Is this text too small for us to read? Is this te oo small for us to read? Is this text too small for us to read? Is this text too sm

to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is

to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is Limited by
 Eyesight

Limited by Physics to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text to small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for u to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is this text too small for us to read? Is

Parallelism

Concurrency At least two threads are making progress. Parallelism At least two threads are executing simultaneously. Oracle Multithreaded Programming Guide http://docs.oracle.com/cd/E19455-01/806-5257/6je9h032b/index.html vs.

Parallelism Keyboard Mouse Disk Time ======> Concurrency Task Switching K M D K M D K M D K M D K M D K M D Time ======>

How?

Embarrassingly parallel problem.

Embarrassingly parallel problem. … like finding prime numbers.

No content

No content

No content

ssh-keygen

authorized_keys

RPi01 RPi02 RPi03 RPi04 authorized_keys

No content

The cluster works!

No content

No content

RPi Summary •Assemble the cluster. •Communicate via mpi4py. •Assign a portion of the problem to each node. •Gather the results.

What’s next?

Graphical Processing Unit

MacBook Pro Video 384 cores

Raspberry Pi is a gateway for the GPU.

No content

No content

No content

No content

No content

No content

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO
 Adapteva

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO
 Adapteva It’s hard.

Mapping complicated algorithms to massively parallel hardware architectures is considered a non-trivial problem. - Andreas Olofsson, CEO
 Adapteva It’s hard. And it’s worth it.

Moore’s Law: 2x every 18 months

Moore’s Law for GPUs https://www.nextbigfuture.com/2017/04/for-commoners-using-cpus-meaningful.html Moore Nvidia

# of Nodes Total Render Time Power Cost (5yrs) System Cost CPU - Dual Skylake 25 38 hours $70k $250k Nvidia RTX Server 1 6 hours $10k $30k “Pixar artists already rely on Nvidia ray tracing, and RTX more than doubles the performance they will see. We’re excited to use RTX on our upcoming films.” -Steve May, CTO, Pixar From Jensen Huang’s keynote at Nvidia GTC 2019

Nvidia Jetson Nano 128 GPU cores 4 CPU cores

Nvidia Jetson Nano 128 GPU cores 4 CPU cores $99

Nvidia Jetson Nano 128 GPU cores 4 CPU cores $99

No content

Compute Unified Device Architecture

CUDA supports Python, Clojure, C, C++, C#, Fortran, Ruby, Java…

No content

No content

Embarrassingly parallel problem. … deep learning.

No content

No content

•Training •Testing •Deployment

Avoiding “gotchas” •Lots of data/examples •Labeling. •Randomly choose 20% for testing. •Don’t train w/testing data.

Parallelism accelerates deep learning.

No content

No content

No content

Resources

No content

https://www.youtube.com/watch?v=aircAruvnKk&list=PLOnWKC1gI_OOIrkRZDq2SgYoDxfRAJtdF&index=66&t=0s Recommended by the Nvidia instructors.

http://alexlenail.me/NN-SVG/index.html

No content

No content

No content

No content

Summary… •Parallelism + RPi •Jetson Nano •Deep Learning •Resources

In business, there is nothing more valuable than a technical advantage your competitors don’t understand. -Paul Graham http://www.paulgraham.com/avg.html

Thanks! RayHightower.com