Share
Embed
Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
2
Slide 3
Slide 3 text
Chapter 1.
Icebreaker
Have you ever seen anything like this?
3
Slide 4
Slide 4 text
Chapter 1.
Icebreaker
Colors look
Have you ever seen anything like this? washed out
( ⾊が淡い)
Replace with the broken photo
later
4
Slide 5
Slide 5 text
Chapter 1.
Icebreaker
Have you ever seen anything like this? The glow
disappears
(輝きが消える)
5
Slide 6
Slide 6 text
Chapter 1.
Icebreaker
Have you ever seen anything like this?
Blown-out
highlights
(⽩⾶び)
6
Slide 7
Slide 7 text
Chapter 1.
Icebreaker
Colors look
Have you ever seen anything like this? washed out
The glow
disappears
Blown-out
highlights
7
Slide 8
Slide 8 text
Chapter 1.
Icebreak
Colors turn
Have you ever seen anything like this? washed-out
It's NOT a problem on
The glow
the receiving side
disappears
Blown-out
highlights
8
Slide 9
Slide 9 text
Chapter 1.
Icebreak
Colors turn
Have you ever seen anything like this? washed-out
Today's goal: be able
to explain these
The glow
disappears
phenomena
Blown-out
highlights
9
Slide 10
Slide 10 text
Chapter 1.
About Me
Kazuki CHIGITA / @chigichan24
Building the family photo-sharing app
「家族アルバム みてね」(FamilyAlbum) at MIXI, Inc.
New roles just opened.
Weʼre looking for new teammates.
https://mixigroup-recruit.mixi.co.jp/recruitment
-category/career/14287/
10
Slide 11
Slide 11 text
Chapter 1.
Notices
- For clarity, this talk uses the following terms
consistently unless noted otherwise:
Images
(静⽌画)
Still images
(画像)
Photos
(写真)
11
Slide 12
Slide 12 text
Chapter 1.
Notices
- For clarity, this talk uses the following terms
consistently unless noted otherwise:
Images
(静⽌画)
Still images
(画像)
Moving
images
(動画像)
Photos
(写真)
Videos
(動画)
12
Slide 13
Slide 13 text
Chapter 1.
Notices
- For clarity, this talk uses the following terms
consistently unless noted otherwise:
Media
（メディア）
Images
(静⽌画)
Still images
(画像)
Moving
images
(動画像)
Photos
(写真)
Videos
(動画)
13
Slide 14
Slide 14 text
Chapter 1.
Notices (2)
- Theory of video and imaging (HDR included) is mostly
skipped, except where it aids understanding or is
Android-specific.
- These resources are great for the specs and the math:
- https://www.eizo.co.jp/eizolibrary/color_management/hd
r/index.html
- https://www.eizo.co.jp/eizolibrary/other/itmedia02_01/
- https://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/
WHP309.pdf
14
Slide 15
Slide 15 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
15
Slide 16
Slide 16 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
16
Slide 17
Slide 17 text
Chapter 2. What is HDR?
17
Slide 18
Slide 18 text
Chapter 2.
What is HDR?
SDR
HDR
18
Slide 19
Slide 19 text
Chapter 2.
What is HDR?
SDR
HDR
High Dynamic Range
19
Slide 20
Slide 20 text
Chapter 2.
What is HDR?
SDR
Standard Dynamic
Range
HDR
High Dynamic Range
20
Slide 21
Slide 21 text
Chapter 2.
What is HDR?
SDR
HDR
HDR photos and videos can represent
a much wider range of brightness
メディアはより幅広い明るさを表現できます
21
Slide 22
Slide 22 text
Chapter 2.
What is HDR?
SDR
HDR
22
Slide 23
Slide 23 text
Chapter 2.
What is HDR
What is HDR?
SDR
HDR
HDRとは？
A technology for representing
a wider range of brightness
明るさを良い感じにする技術
23
Slide 24
Slide 24 text
Chapter 2.
Note: many different things are called
"HDR"
24
Slide 25
Slide 25 text
Chapter 2.
Note: many different things are called
"HDR"
HDR+
HDR10
HLG
HDR10+
Smart HDR
Ultra HDR
Live HDR+
Dolby Vision
Super HDR
Auto HDR
25
Slide 26
Slide 26 text
Chapter 2.
Note: many different things carry the
name "HDR"
HDR+
HDR10
HLG
Hmm…?
HDR10+
Smart HDR
Ultra HDR
Live HDR+
Dolby Vision
Super HDR
Auto HDR
26
Slide 27
Slide 27 text
Chapter 2.
HDR standards and features
HDR+
HDR10
HLG
HDR10+
Smart HDR
Ultra HDR
Live HDR+
Dolby Vision
Super HDR
Auto HDR
27
Slide 28
Slide 28 text
Chapter 2.
HDR standards and features
HDR+
HDR10
HLG
HDR10+
Smart HDR
Ultra HDR
Live HDR+
Dolby Vision
Super HDR
Auto HDR
28
Slide 29
Slide 29 text
Chapter 2.
HDR standards and features
Industry and brand specs
(業界仕様・ブランド名)
Camera feature names
(カメラの機能名称)
HDR10
HDR+
HDR10+
Live HDR+
Dolby Vision
Smart HDR
HLG
Auto HDR
Ultra HDR
Super HDR
29
Slide 30
Slide 30 text
Chapter 2.
HDR standards and features
Industry and brand specs
(業界仕様・ブランド名)
HDR10
Pixel
Camera feature names
(カメラの機能名称)
HDR+
HDR10+
Live HDR+
Dolby Vision
Smart HDR
HLG
Ultra HDR
iPhone
Galaxy
Auto HDR
Super HDR
30
Slide 31
Slide 31 text
Chapter 2.
Note: Pixelʼs HDR+ is not an
Android-specific feature
https://research.google/blog/hdr-with-bracketing-on-pixel-phones/
31
Slide 32
Slide 32 text
Chapter 2.
The specs around HDR
Industry and brand specs
(業界仕様・ブランド名)
HDR10
HDR10+
Dolby Vision
HLG
Ultra HDR
32
Slide 33
Slide 33 text
Chapter 2.
The specs around HDR
Industry and brand specs
(業界仕様・ブランド名)
HDR10
HDR10+
Dolby Vision
HDR video specs
HLG
Ultra HDR
HDR photo specs
33
Slide 34
Slide 34 text
Chapter 2.
The specs around HDR
Industry and brand specs
-
HDR10
HDR10+
Dolby Vision
HLG
Ultra HDR
Galaxy Z Flip 5
HDR video specs
HDR photo specs
Pixel 10
34
Slide 35
Slide 35 text
Chapter 2.
The specs around HDR
Industry and brand specs
-
HDR10
HDR10+
Dolby Vision
HLG
Ultra HDR
Galaxy Z Flip 5
HDR video specs
HDR photo specs
Pixel 10
35
Slide 36
Slide 36 text
Chapter 2.
The specs around HDR
Industry and brand specs
(業界仕様・ブランド名)
HDR10
HDR10+
Dolby Vision
HDR video specs
HLG
Ultra HDR
HDR photo specs
36
Slide 37
Slide 37 text
Chapter 2.
HDR video
What qualifies a
file as HDR
video?
37
Slide 38
Slide 38 text
Chapter 2.
HDR video
What qualifies a
file as HDR
video?
On Android: its transfer function is one of the following
(伝達関数が以下のいずれかであることがHDRの条件)
PQ
HLG
38
Slide 39
Slide 39 text
Chapter 2.
What is a transfer function(伝達関数)?
A function that maps brightness levels to encoded values in a
file (輝度をファイルの数値とどう対応させるかの関数)
39
Slide 40
Slide 40 text
Chapter 2.
What is a transfer function(伝達関数)?
A function that maps brightness levels to encoded values in a
file (輝度をファイルの数値とどう対応させるかの関数)
Eyes see finer differences in the
dark
⼈の⽬は暗い部分の差に敏感
40
Slide 41
Slide 41 text
Chapter 2.
What is a transfer function(伝達関数)?
A function that maps brightness levels to encoded values in a
file (輝度をファイルの数値とどう対応させるかの関数)
Eyes see finer differences in the
dark
⼈の⽬は暗い部分の差に敏感
41
Slide 42
Slide 42 text
Chapter 2.
What is a transfer function(伝達関数)?
A function that maps brightness levels to encoded values in a
file (輝度をファイルの数値とどう対応させるかの関数)
42
Slide 43
Slide 43 text
Chapter 2.
What is a transfer function?
Media3 checks whether content is HDR with code like this
-/ ColorInfo.java
public static boolean isTransferHdr(
@Nullable ColorInfo colorInfo
) {
return colorInfo -= null
-& (colorInfo.colorTransfer -= C.COLOR_TRANSFER_HLG
-| colorInfo.colorTransfer -= C.COLOR_TRANSFER_ST2084);
}
https://github.com/androidx/media/blob/2bc207851df311340767e913931ca7b2
8cab1794/libraries/common/src/main/java/androidx/media3/common/ColorInfo.
java#L316-L326
43
Slide 44
Slide 44 text
Chapter 2.
HDR video
What qualifies a
file as HDR
video?
On Android: its transfer function is one of the following
(伝達関数が以下のいずれかであることがHDRの条件)
PQ
HLG
44
Slide 45
Slide 45 text
Chapter 2.
HDR video
What makes a
file a candidate
for HDR video?
Notices that neither
constant includes ʻHDRʼ
On Android: its transfer function is one of the
following
PQ
HLG
45
Slide 46
Slide 46 text
Chapter 2.
HDR video
What qualifies a
file as HDR
video?
On Android: its transfer function is one of the following
(伝達関数が以下のいずれかであることがHDRの条件)
PQ
HLG
46
Slide 47
Slide 47 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
with various metadata formats.
Proposed by Dolby Laboratories
HLG
Designed to stay backward compatible with SDR.
Needs no metadata. Proposed by NHK and BBC
47
Slide 48
Slide 48 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
with various metadata formats.
Proposed by Dolby Laboratories
HLG
Designed to stay backward compatible with SDR.
Needs no metadata. Proposed by NHK and BBC
48
Slide 49
Slide 49 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
with various metadata formats.
Proposed by Dolby Laboratories
PQ
⼈間の知覚の限界を考慮して設計されたもの
複数のメタデータで情報を補完する。Dolbyが提案
49
Slide 50
Slide 50 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
1. Luminance of the brightest pixel
with various metadata formats.
2. Maximum frame-average
luminance
Proposed by Dolby Laboratories
PQ
⼈間の知覚の限界を考慮して設計されたもの
複数のメタデータで情報を補完する。Dolbyが提案
50
Slide 51
Slide 51 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
1. Luminance of the brightest pixel
with various metadata formats.
2. Maximum frame-average
luminance
Proposed by Dolby Laboratories
PQ
⼈間の知覚の限界を考慮して設計されたもの
複数のメタデータで情報を補完する。Dolbyが提案
And more, depending on
the situation
51
Slide 52
Slide 52 text
Chapter 2.
What is a transfer function(伝達関数)?
HLG
SDRと互換性を持ちながら使⽤できるように設定
PQのようにメタデータが不要で、SDRとしてもつかえる
NHK・BBCが作成
HLG
Designed to stay backward compatible with SDR.
Needs no metadata. Proposed by NHK and BBC
52
Slide 53
Slide 53 text
Chapter 2.
What is a transfer function(伝達関数)?
HLG
SDRと互換性を持ちながら使⽤できるように設定
PQのようにメタデータが不要で、SDRとしてもつかえる
NHK・BBCが作成
HLG
Designed to stay backward compatible with SDR.
Needs no metadata. Proposed by NHK and BBC
53
Slide 54
Slide 54 text
Chapter 2.
What is a transfer function(伝達関数)?
PQ
Built around the limits of human perception. Comes
with various metadata formats.
Proposed by Dolby Laboratories
HLG
Designed to stay backward compatible with SDR.
Needs no metadata. Proposed by NHK and BBC
54
Slide 55
Slide 55 text
Chapter 2.
What is a transfer function?
PQ
So how do specs like
Proposed by Dolby Laboratories
HDR10
relate?
HLG
Built around the limits of human perception. Comes
with various metadata formats.
Designed to stay backward compatible with SDR. Needs no metadata.
Proposed by NHK and BBC
55
Slide 56
Slide 56 text
Chapter 2.
What the specs really are
Spec(規格)
Transfer
Function
(伝達関数)
metadata
(メタデータ)
Device default
撮影時
(標準カメラアプリ)
HDR10
PQ
Static
-
HDR10+
PQ
Dynamic
some Galaxy and
Xiaomi devices
Dolby Vision
PQ/HLG
Dynamic
iPhone, OPPO,
some Xiaomi devices
HLG
HLG
-
Pixel, Xperia,
some Galaxy devices 56
Slide 57
Slide 57 text
Chapter 2.
What the specs really are
Depending on the device ̶
"HDR support" spans many
specs, and misreading
them leads to...
Spec(規格)
Transfer
Function
(伝達関数)
metadata
(メタデータ)
Device default
撮影時
(標準カメラアプリ)
HDR10
PQ
Static
-
HDR10+
PQ
Dynamic
a part of Galaxy, a
part of Xiaomi
DolbyVision
PQ/HLG
Dynamic
iPhone, OPPO,
a part of Xiaomi
HLG
HLG
-
Pixel, Xperia,
a part of Galaxy
57
Slide 58
Slide 58 text
Chapter 2.
HDR photos (Ultra HDR)
A second JPEG is concatenated after the end marker of an ordinary
SDR JPEG. That is the gain map ̶ a map of where and how much to
brighten
Quoted from https://developer.android.com/media/platform/hdr-image-format
58
Slide 59
Slide 59 text
Chapter 2.
HDR photos (Ultra HDR)
59
Slide 60
Slide 60 text
Chapter 2.
HDR photos (Ultra HDR)
Source JPEG
Gain map
60
Slide 61
Slide 61 text
Chapter 2.
Why care about HDR on Android?
Let's unpack a bit of history!
Video
Photos
61
Slide 62
Slide 62 text
Chapter 2.
Why care about HDR on Android?
Let's unpack a bit of history!
Video
Photos
62
Slide 63
Slide 63 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
2014
2016
2019
2022
2026
63
Slide 64
Slide 64 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
Two major families of
specs take shape (PQ,
HLG)
2014
2016
2019
2022
2026
64
Slide 65
Slide 65 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
Android 7.0 lands the
APIs: HDR video playback
supported
2014
2016
2019
2022
2026
65
Slide 66
Slide 66 text
Chapter 2.
Why care about HDR on Android?
HDR10 (PQ)
History of HDR video
DolbyVision (PQ)
HLG
Android 7.0 lands
the
APIs: HDR video playback
supported
and more, supported at this point
only for PLAY.
2014
2016
2019
2022
2026
66
Slide 67
Slide 67 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
Android 10 adds HDR10+
̶ the lineup catches up
with today's
2014
2016
2019
2022
2026
67
Slide 68
Slide 68 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
Android 13 makes
HLG10 support
mandatory
2014
2016
2019
2022
2026
68
Slide 69
Slide 69 text
Chapter 2.
Why care about HDR on Android?
History of HDR video
Android 17 starts
supporting Eclipsa
Video
2014
2016
2019
2022
2026
69
Slide 70
Slide 70 text
Chapter 2.
Why care about HDR on Android?
Let's unpack a bit of history!
Video
Photos
70
Slide 71
Slide 71 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
2020
2023
2024
2025
71
Slide 72
Slide 72 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
Apple introduces HDR
photos with iPhone 12
2020
2023
2024
2025
72
Slide 73
Slide 73 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
Apple introduces HDR
photos with iPhone 12
Adobe brings its own
flavor of HDR
2020
2023
2024
2025
73
Slide 74
Slide 74 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
Android 14 introduces
Ultra HDR
Apple introduces HDR
photos with iPhone 12
Adobe brings its own
flavor of HDR
2020
2023
2024
2025
74
Slide 75
Slide 75 text
Chapter 2.
Why care about HDR on Android?
Apple, Adobe and
Google each push
photo HDR with their own
History of HDR photos
Apple introduces HDR
photos with iPhone 12
Android 14 introduces
Ultra HDR
Adobe brings its own
flavor of HDR
gain map formats
2020
2023
2024
2025
75
Slide 76
Slide 76 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
A move toward one
unified spec
2020
2023
2024
2025
76
Slide 77
Slide 77 text
Chapter 2.
Why care about HDR on Android?
ISO-compatible export
supported from
Android 15
History of HDR photos
A move toward one
unified spec
2020
2023
2024
2025
77
Slide 78
Slide 78 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
The long-awaited ISO
standardization
2020
2023
2024
2025
78
Slide 79
Slide 79 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
ISO 21496-1:2025
Digital photography ̶
Gain map
The long-awaited
standardization
metadata for imageISOconversion
2020
2023
2024
2025
79
Slide 80
Slide 80 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
The long-awaited
ISO standardization
2020
2023
2024
2025
80
Slide 81
Slide 81 text
Chapter 2.
Why care about HDR on Android?
History of HDR photos
HEIC support from
Android 16
The long-awaited
ISO standardization
2020
2023
2024
2025
81
Slide 82
Slide 82 text
Chapter 2.
Chapter 2. Wrap-up
➢ These past few years, user environments went from merely
playing HDR media to generating it.
➢ Meanwhile ̶ for video especially ̶ mishandling generated
HDR looks broken through the user's eyes.
➢ The range of support differs by OS version, device, and
library.
Preventing this kind of breakage is the app developer's
responsibility
82
Slide 83
Slide 83 text
Chapter 2.
Chapter2. まとめ
HDRは従来の、この数年でユーザーの環境でHDRメディアを再⽣す
るだけでなく、⽣成できるような環境が整ってきた。
➢ ⼀⽅で、特に動画では⽣成されたHDR動画の扱いを誤ると
ユーザーの⽬線からは「壊れて」いるようにみえる。
➢ OSバージョンや、デバイス、ライブラリによってサポートの範囲が
異なる。
➢
「壊れて 」しまわないように
適切にハンドリングするのがアプリ開発者の役⽬
83
Slide 84
Slide 84 text
Chapter 2.
Chapter2. Timeline Wrap-up
Android 7
10
13
14
84
Slide 85
Slide 85 text
Chapter 2.
Chapter2. Timeline Wrap-up
Be able to
play HDR
video
Android 7
10
13
14
85
Slide 86
Slide 86 text
Chapter 2.
Chapter2. Timeline Wrap-up
Be able to
play HDR
video
Android 7
Generate
HDR video
(opt-in)
10
13
14
86
Slide 87
Slide 87 text
Chapter 2.
Chapter2. Timeline Wrap-up
Be able to
play HDR
video
Android 7
Generate
HDR video
(opt-in)
10
Generate
HDR video
(API support)
13
14
87
Slide 88
Slide 88 text
Chapter 2.
Chapter2. Timeline Wrap-up
Be able to
play HDR
video
Android 7
Generate
HDR video
(opt-in)
10
Generate
HDR video
(API support)
13
14
Generate /
Confirm HDR
image
88
Slide 89
Slide 89 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
89
Slide 90
Slide 90 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
90
Slide 91
Slide 91 text
Chapter 3. Case 1: washed-out
colors and ColorSpace
91
Slide 92
Slide 92 text
Chapter 3.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
92
Slide 93
Slide 93 text
Chapter 3.
How HDR media reaches the user
Display
Edit
/
Real-world
issues
I
Upload
in app
process
encoded, mapped to
Thumbna
Receive
Display
this
pipeline
ils
media
in app
Capture
Ch. 4/5
Ch. 3/5
Ch. 3
Ch. 5
Ch. 3
Ch.
3/4/5
93
Slide 94
Slide 94 text
Chapter 3.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
94
Slide 95
Slide 95 text
Chapter 3.
Case 1: colors look washed out(淡い⾊)
Expected photo
Actual photo
95
Slide 96
Slide 96 text
Chapter 3.
When editing or transcoding
When rewriting a video's format for editing and similar work,
you use these methods
val format = MediaFormat.createVideoFormat(
“video/hevc”,
-/ mime
1920,
-/ width
1080
-/ height
)
https://developer.android.com/reference/android/media/MediaFormat#cr
eateVideoFormat( java.lang.String,%20int,%20int)
96
Slide 97
Slide 97 text
Chapter 3.
When editing or transcoding
When rewriting a video's format for editing and similar work,
you use these methods
val format = MediaFormat.createVideoFormat(
“video/hevc”,
-/ mime
1920,
-/ width
1080
-/ height
Don't forget to pass the HDR
)
parameters here!!
https://developer.android.com/reference/android/media/MediaFormat#cr
eateVideoFormat( java.lang.String,%20int,%20int)
97
Slide 98
Slide 98 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
98
Slide 99
Slide 99 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
Color space
(BT.709, BT.2020)
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
99
Slide 100
Slide 100 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
Transfer function
(ST_2084, HLG)
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
100
Slide 101
Slide 101 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
Color range
(Full, Limited)
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
101
Slide 102
Slide 102 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
HDR10 static
metadata
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
102
Slide 103
Slide 103 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
HDR10+ dynamic
metadata
MediaFormat.KEY_HDR_ST2094_50_INFO
103
Slide 104
Slide 104 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
Metadata for
newer HDR specs
104
Slide 105
Slide 105 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
In the SDR era, leaving these
unset mostly worked ̶ beware
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
MediaFormat.KEY_HDR_ST2094_50_INFO
105
Slide 106
Slide 106 text
Chapter 3.
The HDR-related keys
MediaFormat.KEY_COLOR_STANDARD
MediaFormat.KEY_COLOR_TRANSFER
MediaFormat.KEY_COLOR_RANGE
MediaFormat.KEY_HDR_STATIC_INFO
MediaFormat.KEY_HDR10_PLUS_INFO
Not needed for HLG;
required for PQ support
MediaFormat.KEY_HDR_ST2094_50_INFO
106
Slide 107
Slide 107 text
Chapter 3.
In code it looks like this
private val COLOR_PARAM_LIST = listOf(
KEY_COLOR_STANDARD,
KEY_COLOR_TRANSFER,
KEY_COLOR_RANGE,
)
COLOR_PARAM_LIST.forEach { key ->
if (source.containsKey(key)) {
dest.setInteger(key, source.getInteger(key))
}
}
if (source.containsKey(KEY_HDR_STATIC_INFO) {
dest.setByteBuffer(KEY_HDR_STATIC_INFO, getSource())
}
107
Slide 108
Slide 108 text
Chapter 3.
Chapter 3. Wrap-up
➢ SDR時代では気にする必要のなかったパラメータがある。
➢ コンパイルエラーのような形では気づくことは困難なので、実際の画
⾯をチェックするしかない。症状別に以下のようなミスが⼀般的
○ なぜか⾊が淡くなる (⾊空間のパラメータ落ち)
○ ⽩っぽくなる (伝達関数のパラメータ落ち)
○ 輝きがなくなる (メタデータのパラメータ落ち)
再エンコードのような処理をするときには
必ずパラメータをコピする！
108
Slide 109
Slide 109 text
Chapter 3.
Chapter 3. Wrap-up
➢ There are more parameters than the SDR era ever made you
think about
➢ No compile error will catch them ̶ they surface as
symptoms:
○ Somehow washed-out (missing color space)
○ Whitish (missing transfer function)
ちなみにmedia3はこのあたりをサポートしています
○ The glow is gone (missing metadata)
media3's Transformer takes care of
this for you
When editing or re-encoding video,
copy the parameters it needs
109
Slide 110
Slide 110 text
Chapter 3.
Chapter 3. Wrap-up
➢ There are more parameters than the SDR era ever made you
think about
➢ No compile error will catch them ̶ they surface as
symptoms:
○ Somehow washed-out (missing color space)
○ Whitish (missing transfer function)
再エンコード操作はできるならばライブラリに頼りましょう
○ The glow is gone (missing metadata)
Don't hand-roll it ̶ use a library
you can trust
When editing or re-encoding video,
copy the parameters it needs
110
Slide 111
Slide 111 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
111
Slide 112
Slide 112 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
112
Slide 113
Slide 113 text
Chapter 4. Case 2: blown-out
highlights and tone mapping
113
Slide 114
Slide 114 text
Chapter 4.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
114
Slide 115
Slide 115 text
Chapter 4.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
115
Slide 116
Slide 116 text
Chapter 4.
Highlights blow out to white (⽩⾶び)
Expected photo
Actual photo
116
Slide 117
Slide 117 text
Chapter 4.
(Again) pass the parameters properly
➢ First, per Chapter 3's conclusion, suspect missing
parameters
○ Somehow washed-out (missing color space)
○ Whitish (missing transfer function)
○ The glow is gone (missing metadata)
117
Slide 118
Slide 118 text
Chapter 4.
(Again) pass the parameters properly
➢ First, per Chapter 3's conclusion, suspect missing
parameters
○ Somehow washed-out (missing color space)
○ Whitish (missing transfer function)
○ The glow is gone (missing metadata)
And yet, sometimes the video may still look slightly
washed out
118
Slide 119
Slide 119 text
Chapter 4.
The same HDR video looks different
across devices
Device A
Device B
119
Slide 120
Slide 120 text
Chapter 4.
The same HDR video looks different
across devices
Exactly the same file,
a different look
Device A
Device B
120
Slide 121
Slide 121 text
Chapter 4.
The same HDR video looks different
across devices
- Because tone mapping is vendor-dependent.
トーンマッピングの処理がベンダー依存
- PQ lacks HLG-style backward compatibility, so failed tone
mapping can blow out highlights
PQにはHLGのような後⽅互換がないので、トーンマッピングに
失敗すると⽩⾶びが起きやすい
- And the specs adapt to the luminance each device can
express ̶ display differences are inevitable by design
そもそもデバイスの表⽰可能な輝度に合わせてdynamicに決めるので
表⽰差が出るのは当たり前
121
Slide 122
Slide 122 text
Chapter 4.
The same HDR video looks different
across devices
- Because tone mapping is vendor-dependent.
トーンマッピングの処理がベンダー依存
- PQ lacks HLG-style backward compatibility, so failed tone
mapping can blow out highlights
PQにはHLGのような後⽅互換がないので、トーンマッピングに
失敗すると⽩⾶びが起きやすい
The process that compresses and maps values
- And the specs adapt to the luminance each device can
into the range the device's display can actually
express ̶ display differences
show are inevitable by design
そもそもデバイスの表⽰可能な輝度に合わせてdynamicに決めるので
表⽰差が出るのは当たり前
122
Slide 123
Slide 123 text
Chapter 4.
The same HDR video looks different
across devices
- Because tone mapping is vendor-dependent.
トーンマッピングの処理がベンダー依存
- PQ lacks HLG-style backward compatibility, so failed tone
mapping can blow out highlights
PQにはHLGのような後⽅互換がないので、トーンマッピングに
失敗すると⽩⾶びが起きやすい
- And the specs adapt to the luminance each device can
express ̶ display differences are inevitable by design
そもそもデバイスの表⽰可能な輝度に合わせてdynamicに決めるので
There are levels to how far you take this,
表⽰差が出るのは当たり前
depending on the situation
123
Slide 124
Slide 124 text
Chapter 2.
What the specs really are
Spec(規格)
Transfer
Function
(伝達関数)
metadata
(メタデータ)
Device default
撮影時
(標準カメラアプリ)
HDR10
PQ
Static
-
HDR10+
PQ
Dynamic
some Galaxy and
Xiaomi devices
DolbyVision
PQ/HLG
Dynamic
iPhone, OPPO,
some Xiaomi devices
HLG
HLG
-
Pixel, Xperia,
some Galaxy devices124
Slide 125
Slide 125 text
Chapter 4.
How to reduce the differences between
devices
We have three choices
1. Leave it to the OS
2. Map to SDR
3. Decide your own tone mapping and write a
GL shader
125
Slide 126
Slide 126 text
Chapter 4.
How to reduce the differences between
devices
1. Leave it to the OS
- Accept this as a known issue.
既知の問題として扱う
- From Android 16, a new HDR tone mapping method called the
Look-up Table (LUT) is proposed to minimize it.
Android16からはこの問題を解決するためにLUTというトーンマッピング
がある
126
Slide 127
Slide 127 text
Chapter 4.
How to reduce the differences between
devices
2. Map to SDR
- We lose HDR's expressiveness, but today's environments mostly
align
HDRでの表現⼒は失うが、最低限SDRでは表⽰できるようにする
127
Slide 128
Slide 128 text
Chapter 4.
How to reduce the differences between
devices
3. Decide your own tone mapping and write a GL shader
- Supporting every environment this way is realistically hard
⾃前でトーンマッピング⽅式を決めて、GLシェーダーを書く
128
Slide 129
Slide 129 text
Chapter 4.
How to reduce the differences between
devices
We have three choices
1. Leave it to the OS
2. Map to SDR
3. Decide your own tone mapping and write a
GL shader
129
Slide 130
Slide 130 text
Chapter 4.
Chapter 4. Wrap-up
➢ Even with identical parameters, display problems can follow
from tone mapping differing across devices
➢ Whether you accept HDR's quirks or convert to SDR depends
on the situation
➢ As a last resort, writing a GL shader is an option
Tone mapping has no precise spec + Android is diverse ̶
that's the cause. LUTs are moving it toward resolution.
130
Slide 131
Slide 131 text
Chapter 4.
Chapter4. Wrap-up
➢ パラメータがどんなに同⼀でも端末間でトーンマッピング⽅式が
異なることに起因して表⽰に問題が出ることがある
➢ HDRでの問題を受け⼊れるか、SDRに変換するかは状況に応じて使
い分ける
➢ 最終⼿段としてGLシェーダを書く選択肢もある
トーンマッピングに明確な仕様がない + Androidの多様性によっ
て発⽣している。LUTによって解決に向かいつつある。
131
Slide 132
Slide 132 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
132
Slide 133
Slide 133 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
133
Slide 134
Slide 134 text
Chapter 5. Case 3: the glow and
gain maps
134
Slide 135
Slide 135 text
Chapter 5.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
135
Slide 136
Slide 136 text
Chapter 5.
How HDR media reaches the user
Capture
Display
in app
Edit /
process
Upload
Ch. 4/5
Ch. 3/5
Ch. 3
Thumbna
ils
Receive
media
Display
in app
Ch. 5
Ch. 3
Ch.
3/4/5
136
Slide 137
Slide 137 text
Chapter 5.
Case 3: the glow gets lost
Expected photo
Actual photo
137
Slide 138
Slide 138 text
Chapter 5.
Works on my device until you share it...
- A photo captured as Ultra HDR previews perfectly on
your own device, whatever you do
- The moment you share it to another app or user, the
glow is gone
138
Slide 139
Slide 139 text
Chapter 5.
The gain map disappears
- Ultra HDR stores where and how much to brighten, as
a gain map
(again)
139
Slide 140
Slide 140 text
Chapter 5.
The gain map disappears
- Ultra HDR stores where and how much to brighten, as
a gain map
In almost every glow-gone pattern,
the file is being recognized as SDR
(again)
140
Slide 141
Slide 141 text
Chapter 5.
Experiment 1: typical server-side image
processing
1.
2.
3.
4.
resize/resave with Pillow (a Python library)
resize/resave with ImageMagick
ffmpeg
cp command
141
Slide 142
Slide 142 text
Chapter 5.
Experiment 1: typical server-side image
processing
1. resize/resave with Pillow (a Python library)
142
Slide 143
Slide 143 text
Chapter 5.
Experiment 1: typical server-side image
processing
1. resize/resave with Pillow (a Python library)
2. resize/resave with ImageMagick
143
Slide 144
Slide 144 text
Chapter 5.
Experiment 1: typical server-side image
processing
1. resize/resave with Pillow (a Python library)
2. resize/resave with ImageMagick
3. ffmpeg
144
Slide 145
Slide 145 text
Chapter 5.
Experiment 1: typical server-side image
processing
1.
2.
3.
4.
resize/resave with Pillow (a Python library)
resize/resave with ImageMagick
ffmpeg
cp command
145
Slide 146
Slide 146 text
Chapter 5.
Experiment 1: typical server-side image
processing
1.
2.
3.
4.
resize/resave with Pillow (a Python library)
resize/resave with ImageMagick
ffmpeg
cp command
With everything except the cp command, identification
data like XMP and MPF gets lost on read
146
Slide 147
Slide 147 text
Chapter 5.
Experiment 1: typical server-side image
processing
1.
2.
3.
4.
resize/resave with Pillow (a Python library)
resize/resave with ImageMagick
ffmpeg
cp command
The result: fallback to SDR
With everything except the cp command, identification
data like XMP and MPF gets lost on read
147
Slide 148
Slide 148 text
Chapter 5.
Experiment 2: client-side failure cases
Ultra HDR fails to display properly in these cases:
以下のような問題点により問題が発⽣することもあります
- Forgetting to set colorMode
colorMode の指定漏れ
- hasGainmap diverging from reality
hasGainmap実装の状態の乖離
- Editing Ultra HDR through Glide
Glideによる編集
148
Slide 149
Slide 149 text
Chapter 5.
1. Forgetting to set colorMode
We must explicitly set
window.colorMode to COLOR_MODE_HDR
149
Slide 150
Slide 150 text
Chapter 5.
1. Forgetting to set colorMode
We must explicitly set window.colorMode to
COLOR_MODE_HDR
window.colorMode = if (img.hasGainmap()) {
ActivityInfo.COLOR_MODE_HDR
} else {
ActivityInfo.COLOR_MODE_DEFAULT
}
150
Slide 151
Slide 151 text
Chapter 5.
1. Forgetting to set colorMode
We must explicitly set window.colorMode to
COLOR_MODE_HDR
window.colorMode = if (img.hasGainmap()) {
ActivityInfo.COLOR_MODE_HDR
} else {
ActivityInfo.COLOR_MODE_DEFAULT
}
It can also be set app-wide from the Manifest
151
Slide 152
Slide 152 text
Chapter 5.
2. hasGainmap diverging from reality
When a gain map is present, this method returns true.
Bitmap.hasGainmap()
However, image processing can leave it in a state that
cannot be handled...
152
Slide 153
Slide 153 text
Chapter 5.
2. hasGainmap diverging from reality
When a gain map is present, this method returns true.
Bitmap.hasGainmap()
However, image processing can leave it in a state that
cannot be handled...
Some devices (Galaxy S24) may only support the pre-ISO
header format.
153
Slide 154
Slide 154 text
Chapter 5.
2. Verify: is it really being rendered as HDR?
Recommended: check the Display's hdrSdrRatio like below
→ see how the Display ultimately judged it
private fun logDisplayHdrState() {
val d = display /: return
Timber.i("hdrSdrRatio=${fmt(d.hdrSdrRatio)}")
Timber.i("NOTE: hdrSdrRatio stuck at 1.0
means SDR")
}
154
Slide 155
Slide 155 text
Chapter 5.
3. Editing Ultra HDR through Glide
- With Glide,
- even processing as ordinary as this turns Ultra HDR
into SDR.
- Because centerCrop() goes through a Canvas
Glide.with(context)
.load(uri)
.centerCrop() // the gain map is lost here
.into(imageView)
155
Slide 156
Slide 156 text
Chapter 5.
Chapter 5. Wrap-up
➢ If an Ultra HDR photo loses its glow, It's probably being
treated as SDR
➢ Checkpoints exist on both client and server ̶ check both
➢ Asking how the Display judged it is the last resort
➢ Library support for editing Ultra HDR is thin
On failure the spec falls back to SDR, so the user's intent
is rarely blocked outright!
156
Slide 157
Slide 157 text
Chapter 5.
Chapter5. まとめ
➢
➢
➢
➢
輝きを失った写真はUltra HDRがSDRとして認識されたパターン
クライアント・サーバいずれにも確認ポイントがあるので要チェック
Displayがどう評価したかを確認するのが最終⼿段
ライブラリではUltra HDRの編集のサポートは⼿薄
失敗したときは、SDRにフォールバックされる仕様なので、
ユーザーさんの意図を阻害することは少ない！
157
Slide 158
Slide 158 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
158
Slide 159
Slide 159 text
Contents
Chapter 1. Icebreak
Chapter 2. What is HDR?
Chapter 3. Case 1: washed-out colors and ColorSpace
Chapter 4. Case 2: blown-out highlights and tone mapping
Chapter 5. Case 3: the glow and gain maps
Chapter 6. Wrap-up
159
Slide 160
Slide 160 text
Chapter 6. Wrap-up
160
Slide 161
Slide 161 text
Chapter 6.
Remember today's goal?
161
Slide 162
Slide 162 text
Chapter 1.
Icebreak
Colors turn
Have you ever seen anything like this? washed-out
Today's goal: be able
The glow
to explain these
disappears
phenomena
Photo to be replaced later
Replace with the broken photo
later
Blown-out
highlights
162
Slide 163
Slide 163 text
Chapter 6.
HDR and the patterns of "breaking"
➢ The HDR patterns
○ Photos (Ultra HDR)
○ Video (HDR10, HDR10+, DolbyVision, HLG)
➢ The "breaking" patterns
○ Any parameter that makes up HDR gets dropped
■ Implementation gaps
■ OS versions and devices
■ Library internals
○ The fallback-to-SDR pattern
163
Slide 164
Slide 164 text
Chapter 6.
Devices used for verification
➢ For video especially, different devices adopt different
specs ̶ keep that in mind:
○ Differences in transfer functions
○ Differences in color spaces
○ Vendor-specific extensions and implementations
➢ Also mind which OS versions exhibit the problems.
➢ HLG is becoming the de facto direction, so this may
improve going forward?
164
Slide 165
Slide 165 text
Chapter 6.
The FIRST step
- Receiving side
データ受信側
- Map the parameters correctly at every step
- Sending side
データ送信側
- Start from what keeps backward compatibility
- HLG for video
- Ultra HDR for photos
- When in doubt, deliberately convert to SDR
- Video HDR especially is in a transition period ̶
165
don't take on too much breakage risk
Slide 166
Slide 166 text
166
Slide 167
Slide 167 text
Let's continue the conversation at
Ask the Speaker
167
Slide 168
Slide 168 text
Chapter 6.
Refs.
Android / AOSP 公式
- https://developer.android.com/media/grow/hdr-playback
- https://developer.android.com/media/platform/hdr-image-format
- https://developer.android.com/media/grow/ultra-hdr
https://developer.android.com/blog/posts/eclipsa-video-hdr-that-looks-right-on
-every-screen
- https://source.android.com/docs/core/display/tone-mapping
（libtonemap / LUT / 断⽚化の⼀次ソース）
- https://source.android.com/docs/core/display/hdr
168
Slide 169
Slide 169 text
Chapter 6.
Refs.
規格
- https://www.iso.org/standard/86775.html （ISO 21496-1 gain map）
- https://www.itu.int/rec/R-REC-BT.2100 （PQ / HLG）
ベンダー⼀次
https://www.apple.com/newsroom/2020/10/apple-introduces-iphone-12-pro-and-iphone-12
-pro-max-with-5g/
- https://www.samsung.com/us/support/answer/ANS10003388/
壊れ⽅の実例
- https://github.com/google/libultrahdr
- https://github.com/bumptech/glide/issues/5333
169