FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks

AAAI 2024

Transcript

1. FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks Shahriar Iqbal, Jianhai Su, Lars Kotthoff, Pooyan Jamshidi larsko@uwyo.edu AAAI, 24 February 2024 1
2. One Size Does Not Fit All 1 1.5 2 2.5 3 3.5 ·104 15 20 25 30 35 40 Energy Consumption (mJ) Prediction Error (%) Xception ← Energy consumption varies 4 × → ← Prediction Error varies 3 × → 2
3. Heterogeneous Parameters Num of Filters, Filter Size, Learning Rate, Num of Epochs DN N Design Compiler Hardware Deployment Num of Active CPUs, CPU/ GPU/ EMC Frequency Cloud, IoT, Edge Num of Threads, GPU Threads, Memory Growth 3
4. Cost-Unaware Methods Waste Resources Coupled Unaware Pareto Optimal Prediction Error (%) Log Wall Clock Time Energy Consumption (mJ) 3000 6000 9000 12000 15 25 35 45 3.65 3.50 3.35 Decoupled Aware Pareto Optimal Prediction Error (%) Log Wall Clock Time Energy Consumption (mJ) 3000 6000 9000 12000 15 25 35 45 3.65 3.50 3.35 4
5. Proposed Method ▷ weight expected benefit of evaluation by cost ▷ choose which objective(s) to evaluate ▷ more efficient use of resources – lower cost, more evaluations 5
6. Results – Computer Vision 0 50 100 150 200 Cumulative Log WallClock Time 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error Xception PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 10000 15000 20000 25000 Energy Consumption (mJ) 15 20 25 30 35 40 Prediction Error (%) Xception PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 6
7. Results – NLP 0 50 100 150 200 Cumulative Log WallClock Time 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error BERT-SQuAD PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 70000 80000 90000 Energy Consumption (mJ) 20 25 30 35 Prediction Error (%) BERT-SQuAD PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 7
8. Results – Speech Recognition 0 50 100 150 200 250 300 Cumulative Log WallClock Time 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 Energy Consumption (mJ) 17.5 20.0 22.5 25.0 27.5 30.0 32.5 35.0 Prediction Error (%) DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 8
9. Results – Evaluations 0 20 40 60 80 100 120 140 160 180 200 PAL 0 20 40 60 80 100 120 140 160 180 200 PESMO-DEC 2 4 6 8 0 20 40 60 80 100 120 140 160 180 200 Iteration CA-MOBO 0 20 40 60 80 100 120 140 160 180 200 Iteration FlexiBO 2 4 6 8 9
10. FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks ▷ cost-aware acquisition function decreases cost and improves results ▷ code available at https://github.com/softsys4ai/FlexiBO 0 50 100 150 200 250 300 Cumulative Log WallClock Time 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 Energy Consumption (mJ) 17.5 20.0 22.5 25.0 27.5 30.0 32.5 35.0 Prediction Error (%) DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 10