Optuna: a Black-Box Optimization Framework Preferred Networks, Inc. Hideaki Imamura

2 Hideaki Imamura ● Researcher at Preferred Networks, Inc ● Research on the theory and applications of black-box optimization ● Led Optuna v3.0 and v3.6 ● GitHub: @HideakiImamura

Optuna: a Black-Box Optimization Framework ● Hyperparameters (e.g., learning rate) control behavior of models and algorithms ● Optuna finds the best hyperparameters via trial-and-error 3 ML Model ML Model ML Model

4 Optuna Users in Numbers 10k GitHub stars 16k repositories 3M monthly downloads Used by

5 ● Over 20 maintainers ○ From various companies and universities ● Over 240 developers ● Working with OSS communities & 140+ more contributors Optuna Contributors in Numbers

6 ● Optuna has been globally recognized and seen continuous growth by adapting to user demands ● Collaboration with ML-related OSS and knowledge exchange among users and developers has extended Optuna's reach beyond AI competitions to practical applications ● The latest initiative, OptunaHub, aims to be a platform that disseminates rich, user-generated contents across its rapidly expanding communities Outline: Success and Future of Optuna

7 The Origin of Optuna How did the development start?

8 Development Started as an Internal Tool ● Initially designed for deep learning hyperparameter optimization ● Contributed to the 2nd place in Google AI Open Images Challenge ○ [1809.00778] PFDet: 2nd Place Solution to Open Images Challenge 2018 Object Detection Track Ref: https://www.kaggle.com/c/google-ai-open-images-object-detection-track

9 Optuna Was Released on GitHub as OSS Presentations Documentations Paper (KDD2019) Ref: https://arxiv.org/abs/1907.10902 Ref: https://optuna.org/

10 Why Did We Release Optuna as OSS? Quote from Takuya Akiba, one of the Optuna launch members: "We … place great importance on presenting our study results to contribute to the community. This … will eventually accelerate the advances of the technology … if we can inspire other professionals in the world… This is why we are very active in … releasing software as an OSS.

11 is one of the most important applications of Optuna Early Adopters: ML Competitions Integration with LightGBM Popular Notebook by Grandmaster

12 Our User Community Has Expanded Integrations of many machine learning libraries Medium

13 Industrial Applications of Optuna How have our products been used for real-world problems?

14 Optuna Extends Preferred Networks (PFN) Businesses Materials Science Autonomous Mobile Robot Multimodal Foundation Models

15 Robotics ● PFN is conducting R&D on robotics and selling toC products ● Optuna contributed to accurate robot vision with low latency Autonomous Mobile Robot: kachaka™ (Now on sale in Japan) Company: https://www.pfrobotics.jp/ Kachaka: https://kachaka.life/ Video: https://youtu.be/u_i2X43dBGw

16 Robot Needs Semantic Segmentation ● Semantic segmentation is essential for autonomous robot ● Neural architecture is important for high accuracy and low latency

17 Neural Architecture Search (NAS) by Optuna Lower latency but lower mIoU Higher mIoU but higher latency mIoU Latency (ms) ● Introduced multi-objective optimization into Optuna ● Got better trade-offs of the mIoU and latency compared to the baseline model ● Introduced constrained optimization to limit to the acceptable latency MobileNetV2 (Baseline)

18 Materials Science ● Matlantis™ supports large-scale materials discovery by simulating the behavior of various materials at an atomic level ● Used for energy calculations of material structures Matlantis™ Material Structure (e.g., Ti₂O) Energy 10K Times Faster than conventional methods E( ) E( )

19 Nature Solves an "Annealing" Problem In general, the materials that actually appear in the world have the structure with the lowest energy. Same chemical formula Ti 2 O Has the lowest potential energy Stable structure in nature Has relatively higher potential energy Unstable structure in nature

20 Crystal Structure Prediction (CSP) … Done! Energy: -3.0 First, I want to try this: Structure: Trial 1 … Done! Energy: -1.7 Then, how about this?: Structure: Trial 2 … Done! Energy: -3.3 So, I want to try this: Structure: Trial 3 ● CSP is formulated as a black-box optimization problem ● Input is the structure suggested by Optuna ● Output is the energy calculated by Matlantis

21 Problem: Crystal Structure Prediction (CSP) ~ Part I ~ Ti x O 1-x , x=2/3 → Ti ⅔ O ⅓ = Ti 2 O Same composition (Ti₂O) but different structures yield different energies Better Composition Ratio x

22 Problem: Crystal Structure Prediction (CSP) ~ Part II ~ Better Existing method: Optimize structure for each composition x Composition Ratio x

23 Problem: Crystal Structure Prediction (CSP) ~ Part III ~ Better Objective: Maximize the blue shaded area Optimal energies for each structure Composition Ratio x

24 Problem: Crystal Structure Prediction (CSP) ~ Part IV ~ Better Proposed method: Jointly optimize over all compositions in the multi-objective optimization manner! Composition Ratio x

25 Found New Structures by Our Method Al₂MnCu ● Found new stable structures for multiple elements at various composition ratios ● Achieved good sample efficiency using correlation between structures of different composition ratios New Known Ti₂O Known Ca 3 P 2 New Known New

26 Other Applications Some real-world application examples have been reported by the community. Automated Taxi Dispatch （By Mobility Technololgies at April 2020） Blog Post JVM Parameter Tuning （By UL Systems at June 2021） Ref: https://wiki.openjdk.org/display/zgc/Main Ref: https://www.slideshare.net/slideshow/jvmoptuna-optuna-meetup-1/249496426 Obtained Parameter Default Parameter Better

27 The Future of Optuna How will the ecosystem be?

28 Diverse Needs for Optuna from Real-world Applications Extended problem setup Multi-objective optimization Constrained optimization Batch optimization Multi-fidelity optimization etc… Cutting edge algorithm Algorithms based on the recent academic papers Domain specific method Each real-world application often requires its own unique considerations Incorporating all needs into Optuna is difficult from a software design perspective Benefit the community by sharing assets gained through real-world applications

29 OptunaHub: a Feature-Sharing Platform ● Contributors (researchers and developers) can easily register their own algorithms to OptunaHub ● Users can use those features in Optuna manners

30 OptunaHub is Growing with Contributors Materials Science resnant@Toyota Evolutionary Computation nomuramasahir0@CyberAgent Hyperparameter Optimization [email protected] From the BBO-Rietveld paper under a Creative Commons Attribution 4.0 International License PFNs4BO CatCMA BBORietVeld From https://arxiv.org/abs/2305.17535 From https://arxiv.org/abs/2405.09962

31 Submit a pull request according to the tutorial!!! How to Register Your Algorithm into OptunaHub The top page of OptunaHub The detail page of a feature

32 How to Use OptunaHub $ pip install optunahub Step1: Installation Step2: Load Module and Use! More API details: https://optuna.github.io/optunahub/ Visit hub.optuna.org for the list of all features

33 Wrap up ● Optuna: a globally recognized black-box optimization framework ○ 3M+ downloads/month, 10K+ GitHub stars, 16K+ used in GitHub repos ● Continuous growth by adapting to user demands ○ From Kaggler to ML Practitioners ○ Integrations of ML libraries such as LightGBM and PyTorch ● Industrial Applications ○ Robotics & Materials Science ● Further expansion of Optuna Community ○ New feature-sharing platform OptunaHub $ pip install optuna optunahub

34 We are looking forward to collaborations with you! Visit optuna.org for more information Contact at the GitHub issue/discussions

Making the real world computable