Optimize saliency in specified region in images Mutsuha Asada（University of Tsukuba） Ryo Yonetani（OMRON SINIC X） WCCE2022 １

First of all ・You can play with Focaccia, the library developed in this study, using Google Colaboratory. ・You can access GitHub Gist from 　the QR code on the right and copy 　the Colab Project with "Open in Colab". https://gist.github.com/momeemt/422f4b4dd6cb896183da6e7bf14f1f30 2

3 ・Recently, entertaining video content has been gaining popularity. 　・i.g. YouTube, niconico, TikTok, etc... Background ・What edits can the video editor make to make the target stand out? 　・What kind of filtering would make it stand out?

・Properties that induce bottom-up attention [^1]. ・A fast and easy method to compute saliency maps has been proposed. [^2] 　・Available from OpenCV Contrib saliency.StatusSaliencySpectralResidual_create(). Saliency [1] 上向 俊晃・小峯 一晃・森田 寿哉, 動画像コンテンツにおける注視点マップと顕著性マップとの関係性に関する考察, 情報科学技術フォーラム講演論文集 第8巻 第3号, 2009-08-20 [2] Xiaodi Hou・Liqing Zhang, Saliency Detection: A Spectral Residual Approach, 2007 IEEE Conference on Computer Vision and Pattern Recognition Figure1. the lion Figure2. saliency map in Fig.1 Black is weak, white is strong 4

The goal is to increase the relative saliency of the target area 5 ・We want to increase the saliency of the target area by some filter. ・Or we want to reduce prominence outside of the target area. Definition and Goals For the purposes of this study, we shall define ・saliency = mean value of the image saliency map ・saliency_aoi = average value of saliency map in AOI ・saliency_bg = average value of saliency map for regions excluding AOI

・We would like to think about the parameters of a filter that will increase saliency while keeping the image as unchanged as possible. ・This problem can be solved by minimizing the following objective function f. Objective Function Definition 6

7 ・We would like to think about the parameters of a filter that will increase saliency while keeping the image as unchanged as possible. ・This problem can be solved by minimizing the following objective function f. Objective Function Definition ・Calculate the difference between the saliency (img_AOI) in the area and the saliency (img_bg) outside the area 　→ This value is smaller when the area is more prominent

・Calculate the absolute value of the processed image and the original image multiplied by an appropriate constant λ 　→ This value will be smaller when the image is more unchanged 8 ・Calculate the difference between the saliency (img_AOI) in the area and the saliency (img_bg) outside the area 　→ This value is smaller when the area is more prominent ・We would like to think about the parameters of a filter that will increase saliency while keeping the image as unchanged as possible. ・This problem can be solved by minimizing the following objective function f. Objective Function Definition

・Difficult to predict transitions or search classically because it involves complex functions Minimize 9

・Use the hyper parameter auto-optimization framework Optuna[3]. 　→ Bayesian optimization algorithms can be used to automatically determine hyper parameters. 　→ It can calculate using the history of trials to estimate promising areas. [3] Optuna, https://optuna.org/ Determine filter parameters 10 ・Difficult to predict transitions or search classically because it involves complex functions Minimize

・Difficult to process the entire image to increase the prominence of a region Area Settings 　→ Increase prominence by editing only areas or non-areas ・Unnatural processing when the area is set to a rectangle. 　→ Using superpixels 　・Clustering using K-means 　・Divides an image into small regions in 　the color space. Figure5. Superpixels by SLIC method [^4] [4] https://jp.mathworks.com/help/images/ref/superpixels.html 11

Optimizes Hue and Saturation Hue: 0.3433403669945987, Saturation: 0.2776790417244691 AOI: (250, 300), λ=1.0^(-70) 12 HSV Optimization ① ① Original Image ② Processed Image ③ Difference between ① and ② ④ Saliency map of ① ⑤ Saliency map of ② ⑥ Difference between ④ and ⑤

Optimizes Saturation and Value Saturation: 0.04027819537097488, Value: 0.9245658673096161 AOI: (250, 300), λ=1.0^(-25) 13 HSV Optimization ② ① Original Image ② Processed Image ③ Difference between ① and ② ④ Saliency map of ① ⑤ Saliency map of ② ⑥ Difference between ④ and ⑤

Optimizes Value and Hue Value: 0.7794278867159744, Hue: 0.46981941064640476 AOI: (250, 300), λ=1.0^(-25) 14 HSV Optimization ③ ① Original Image ② Processed Image ③ Difference between ① and ② ④ Saliency map of ① ⑤ Saliency map of ② ⑥ Difference between ④ and ⑤

Optimizes kernel size and sigma kernel_size: 45, sigma: 23.310971137645943 AOI: (250, 300), λ=1.0^(-100) 15 Blur Optimization ④ Saliency map of ① ⑤ Saliency map of ② ⑥ Difference between ④ and ⑤ ① Original Image ② Processed Image ③ Difference between ① and ②

Get people's attention Optimizes Hue, Saturation, and Value H: 0.5134438319360632 S: 0.7217078033442076 V: 0.8240252886447658 AOI: (125, 550), λ=1.0^(-20) 16 ④ Saliency map of ① ⑤ Saliency map of ② ⑥ Difference between ④ and ⑤ ① Original Image ② Processed Image ③ Difference between ① and ②

・Optuna was used to optimize the saliency in a specific region. ・In HSV correction optimization, I was able to increase the saliency of the specified area while maintaining the original image to some extent by dividing the image by superpixels. ・Blurring was able to reduce saliency. ・In the future, we would like to develop our research into the selection of appropriate filters and saliency optimization for video. 17 Conclusion References [1] 上向 俊晃・小峯 一晃・森田 寿哉, 動画像コンテンツにおける注視点マップと顕著性マップとの関係性に関する考察, 情報科学技術フォーラム講演論文集 第8巻 第3号, 2009-08-20 [2] Xiaodi Hou・Liqing Zhang, Saliency Detection: A Spectral Residual Approach, 2007 IEEE Conference on Computer Vision and Pattern Recognition [3] Optuna, https://optuna.org/ [4] https://jp.mathworks.com/help/images/ref/superpixels.html