José M. Franco, Jorge Sevilla and Antonio J. Plaza International Conference on Computational and Mathematical Methods in Science and Engineering Almería, 2013 Parallel implementation of Pixel Purity Index for a GPU cluster 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

Introduction Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Collaborate on the efficient processing of remotely sensed hyperspectral images on HPC architectures 

Objective Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Parallel implementation of a remote sensing algorithm for a GPU cluster Motivation • No previous implementation available • Good results in parallel versions for GPU • Execution time is critical (fire detection, waste control, military target detection,…) 

Hyperspectral images Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Hyperspectral images Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Unmixing problem Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Related to data interpretation • Hyperspectral images contain “mixed” pixels – Value composed by combination of spectrally pure substances or endmembers Involves • Determination of endmembers. • Unmixing “mixed pixels”. 

Endmembers Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Linear mixing model Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Each pixel is modeled as a linear combination of spectrally pure pixels Comportamiento lineal = ∗ Endmembers matrix Material proportion Constraints ≥ 0 and = 1 

Linear mixing model Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

Pixel Purity Index Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Detection of spectrally pure pixels. • Developed by Boardman, Kruse and Green (1993). • Included in commercial software (ENVI). • Only spectral data is considered. • Pixel projections onto random unit vectors (aka. skewers). – Maxima projections pixels are selected 

Pixel Purity Index Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Pixel Purity Index Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • High computational complexity – Calculation of pixel projections. • Easy parallelization – No dependencies among iterations. 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

What is a GPU cluster? Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 A GPU cluster is a computer cluster in which each node is equipped with at least one GPU Cray Xk7 (USA) – Nº 2 TOP 500 

CETA-CIEMAT GPU cluster Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Located at Trujillo (Extremadura, Spain) • Used by researchers in Genomics, HEP, MD, Hyperspectral image analysis, … • Two environments: – Test: 17 nodes w/ 2 x Nvidia Tesla C1060 (Tesla) – Production: 32 nodes w/ 2 x Nvidia Tesla C2050/C2070 (Fermi) 

CETA-CIEMAT GPU cluster Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

CETA-CIEMAT GPU cluster Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

Implementation details Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Communication pattern master-worker – Master node computes – GPUs from each node execute the algorithm • Spatial partitioning of the source image • Mersenne-Twister (GSL library) used for skewer generation 

Implementation details Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Data partition schema Spectral Spatial Hybrid 

Implementation details Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Data partition schema Spectral Spatial Hybrid All spectral values in the same node 

Implementation details Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Implementation details Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

Test description Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Three real images used (AVIRIS): – Cuprite, Nevada (USA)  44 MB – Lake St. Clair, Michigan (USA)  222 MB – Yellowstone, Wyoming (USA)  4 GB • Number of skewers: 15.360 

Test description Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Compared implementations: – Sequential – Computer cluster (CPUs only – 15 nodes) – GPU – GPU cluster (2 GPUs per node – 15 nodes) • All versions compiled with param –O3 

Test description Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Cuprite Lago St. Clair Yellowstone 

Output comparison Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • Test the algorithm in a real case – Mineral detection • Output comparison – ENVI vs. GPU cluster – Is the same group of pixels selected? • Cuprite image – It is widely used as a reference in the field – Location of minerals is well-known 

Output comparison Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 ENVI output GPU cluster output 

Output comparison Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 ENVI output GPU cluster output Kaolinite Alunite Calcite Kaolinite Alunite Calcite 

Performance results Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Yellowstone Sequential Cluster GPU GPU cluster Time (sec.) 83316,717722 5834,629652 N/A 66,799286 Speed-up 1x 14,27x N/A 1262,37x Cuprite Sequential Cluster GPU GPU cluster Time (sec.) 1319,944515 92,411848 25,810000 8,354137 Speed-up 1x 14,28x 51,14x 157,99x Lake St. Clair Sequential Cluster GPU GPU cluster Time (sec.) 5018,109371 345,361966 91,340000 11,197010 Speed-up 1x 14,53x 54,94x 448,17x 

Outline Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Introduction and objective Pixel Purity Index Our GPU cluster Implementation details Results Conclusions and future work 

Conclusions Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • PPI implementation for a GPU cluster – It can be used for real cases. – It is not an alternative to on-board processing (image must be transferred to cluster) – Suitable for large image repository processing • First implementation of a hyperspectral image analysis algorithm for a GPU cluster 

Future work Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 • New optimizations (cuBLAS, skewers generation on GPU, …) • Implementation of other endmember extraction algorithms (OSP, N-FINDR, …) • Implementation of heterogeneous computing techniques (benchmarking) • Performance evaluation vs. OpenMP implementation (multicore CPUs) • Integration on content based image retrieval systems 

Acknowledgments Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 CETA-CIEMAT acknowledges the support received from the European Regional Development Fund through its operational program Knowledge-Based Economy 

Acknowledgments Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 CAPAP-H4 NETWORK 

Thanks for your attention Any question? Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 

Contact info Parallel implementation of Pixel Purity Index for a GPU cluster Franco J.M., Sevilla J. and Plaza A.J. – CMMSE 2013 Authors José M. Franco – [email protected] Jorge Sevilla – [email protected] Antonio J. Plaza – [email protected] Institutions http://www.ceta-ciemat.es http://www.unex.es/investigacion/grupos/hypercomp @CETA_CIEMAT @uexcomunicacion