AAAIWS2020_oral_pptx

,PLJ.BEPOP .BTBZVLJ5BOBLB 0OJTIJ .BTBLJ 5FUTVKJ 0HBXB Scramble Image Classification Using
Adaptation Network "*P5 'FCSVBSZBUd

%FFQ-FBSOJOHXJUIPXOEBUB Use Deep Learning on public area 2 camera smartphone
Take a photo In my own Share photos To the Human Human

%FFQ-FBSOJOHXJUIPXOEBUB Use Deep Learning on public area 3 Human Send
own photos Own face Pets Cloud server Classifier Deep learning

%FFQ-FBSOJOHXJUIPXOEBUB Is this privacy preserved pipeline?? 4 Human Send own
photos Own face Pets Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning

5 Frog Frog(1.00) *EFBMQSJWBDZQSFTFSWFE*NBHF3FDPHOJUJPO Original Image Scrambled Image (privacy protection)

6 Cat Cat(1.00) *EFBMQSJWBDZQSFTFSWFE*NBHF3FDPHOJUJPO Original Image Scrambled Image (privacy protection)

7 Automobile Automobile(0.82) *EFBMQSJWBDZQSFTFSWFE*NBHF3FDPHOJUJPO Original Image Scrambled Image (privacy protection)

8 Bird Bird(1.00) *EFBMQSJWBDZQSFTFSWFE*NBHF3FDPHOJUJPO Original Image Scrambled Image (privacy protection)

%FFQ-FBSOJOHXJUIPXOEBUB Image Pixel shuffling : human non-understandable 9 Human Send
own photos Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning

%FFQ-FBSOJOHXJUIPXOEBUB Image Pixel shuffling : human non-understandable 10 Human Send
own photos Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning Scrambled Image

%FFQ-FBSOJOHXJUIPXOEBUB Adaptation Net: machine understandable 11 Human Send own photos
Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning Adapt. Net Scrambled Image

4DSBNCMF*NBHF$MBTTJGJDBUJPO Goal 1. Training the classifier with high accuracy 2.
Protecting the visual information (Privacy) of image - scramble : shuffle pixels / reverse pixel value 12 Original Image Scrambled Image (privacy protection) Image Scramble

3FMBUFE8PSL 5BOBLB 13 1 2 3 4 1 2 3
4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1. shuffling 2 2 3 4 1 4 1 4 3 1 3 2 2 2 3 4 1 4 1 4 3 1 3 2 3 1 3 2 1 4 1 4 2 2 3 4 Rearrange intensity values In a row Rearrange intensity values As an RGB image 2. Reverse several values -1 -1 -1 -1 Block-wise operation B x B sized block input output Block-wise Image Scramble (same key in each block)

3FMBUFE8PSL 5BOBLB 14 1 2 3 4 1 2 3
4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1. shuffling 2 2 3 4 1 4 1 4 3 1 3 2 2 2 3 4 1 4 1 4 3 1 3 2 3 1 3 2 1 4 1 4 2 2 3 4 Rearrange intensity values In a row Rearrange intensity values As an RGB image 2. Reverse several values -1 -1 -1 -1 Block-wise operation B x B sized block input output

3FMBUFE8PSL "EBQUBUJPO/FUXPSL 5BOBLB Goal - "EBQUBUJPOOFUXPSLFYUSBDUTGFBUVSFNBQGSPN TDSBNCMFEJNBHFT 15 3 1
3 2 1 4 1 4 2 2 3 4 input Classifier CNN Adaptation Network one CNN layer Kernel : B x B (ex) ResNet VGG Block-Wise Feature Extraction

$POUSJCVUJPOT 16 1. 1SPQPTF/FX*NBHF4DSBNCMFNFUIPE Object : Increasing the security level
2. 1SPQPTF/FX"EBQUBUJPO/FUXPSL Object : Classify New Scrambling Image with high accuracy.

1SPQPTFE*NBHF4DSBNCMF.FUIPE 17 1. Extensive Learnable Encryption(ELE) - Consists of two
scrambling methods Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration

1SPQPTFE*NBHF4DSBNCMF.FUIPE 18 1.1 Block Shuffling - Divide Block & shuffle
location Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration

1SPQPTFE*NBHF4DSBNCMF.FUIPE 19 1.2 #MPDLXJTFTDSBNCMFPQFSBUJPO - Shuffle pixels with different keys
in each blocks Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration

1SPQPTFE"EBQUBUJPO/FUXPSL 20 2. Adaptation Network for ELE Block-wise Sub-network Permutation
Matrix Classifier Proposed Parts

1SPQPTFE"EBQUBUJPO/FUXPSL 21 2.1 Block-wise Sub-network - support Block-Wise Pixel Operation
Block-wise Sub-network Permutation Matrix Classifier Block-Wise Pixel Operation

1SPQPTFE"EBQUBUJPO/FUXPSL 22 2.2 Permutation Matrix - support Block Shuffling Block-wise
Sub-network Permutation Matrix Classifier Block Shuffling

1SPQPTFE"EBQUBUJPO/FUXPSL 23 2.1 Block-wise Sub-network - : CNN (kernel size
== block size) image segmentation (・, ) (・, ) ・・・・・・ (・, -) (・, ) ・・・ Input (32x32x3) blocks (4x4x3) x 64 feature (1x1x16) x 64

1SPQPTFE"EBQUBUJPO/FUXPSL 24 2.2 Permutation Matrix - Learning Block Shuffling Output
/ Block Shuffling Input

1SPQPTFE"EBQUBUJPO/FUXPSL 25 2.2 Permutation Matrix - Learning Block Shuffling by
FC Layer ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (-) Output Input /

&YQFSJNFOU 26 0CKFDU ɾ&WBMVBUF4FDVSJUZBOE*NBHFSFDPHOJUJPOMFWFM &WBMVBUJPO ɾ,FZTQBDFJOTDSBNCMFEJNBHF ɾ$MBTTJGJDBUJPOBDDVSBDZPG4DSBNCMJOHJNBHFT 8JUIUIF"EBQUBUJPO/FUXPSL

&WBMVBUJPO4DSBNCMFE*NBHFT 27 Plain LE EtC ELE (proposed) Image Block-Wise key
Same different different Block-Wise Pixel Shuffle ✔ ✔ Block Shuffling ✔ ✔

&WBMVBUJPO"EBQUBUJPO/FUXPSL 28 &WBMVBUFE"EBQUBUJPO/FUXPSL ɾ/P"EBQUBUJPO/FUXPSL EJSFDUMZJOQVUTDSBNCMFJNBHFUPUIFDMBTTJGJFS ɾ-&"EBQUBUJPO/FUXPSL #MPDLXJTF4VCOFUXPSLDPOWMBZFS ɾ&-&"EBQUBUJPO/FUXPSL
QSPQPTFE #MPDLXJTF4VCOFUXPSLPGCMPDLTDPOWMBZFS 1FSNVUBUJPO.BUSJY

3FTVMU4FDVSJUZ&WBMVBUJPO 29 Plain LE EtC ELE (proposed) Image Key Space
0 O(10566) O(10758) O(1055856)

3FTVMU4FDVSJUZ&WBMVBUJPO 30 Plain LE EtC ELE (proposed) Image Key Space
0 O(10566) O(10758) O(1055856) ELE(Proposed Scrambled Image)’s Key Space Become very large ↓ Increase the Security Level !

3FTVMU$MBTTJGJDBUJPO"DDVSBDZ 31 Dataset Adaptation Net Plain image LE EtC ELE
(Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97%

(Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97% Proposed Adaptation Network is not always effective (Plain Image & LE Image results)

(Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97% Proposed Adaptation Network is effective → Different key on blocks and Block Shuffling

4VNNBSZ 34 0CKFDU *ODSFBTFTFDVSJUZPGTDSBNCMFEJNBHF $MBTTJGZTDSBNCMFEJNBHFXJUIIJHIBDD 1SPQPTFE.FUIPE &-&4DSBNCMFE*NBHF
/FX"EBQUBUJPO/FUXPSL 3FTVMUT *ODSFBTFTFDVSJUZ-FWFM &YUSBDUGFBUVSFGSPN4DSBNCMFEJNBHF CZ"EBQUBUJPO/FUXPSL

Thank you! Welcome Questions!

1SPQPTFE"EBQUBUJPO/FUXPSL 36 2.2 Permutation Matrix Problem : Difficult to be
a discrete value ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) input output

1SPQPTFE"EBQUBUJPO/FUXPSL 37 2.2 Permutation Matrix Problem : Difficult to be
a discrete value Solution : Learning with Regularization ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) input output

1SPQPTFE"EBQUBUJPO/FUXPSL 38 * Regularization 5-7 Penalty : on FC Layer
SQBUJBMTNPPUIOFTTQFOBMUZ: on output feature : ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) 5-7 Penalty spatial smoothness penalty Combined Feature Pixel shuffle :

1SPQPTFE"EBQUBUJPO/FUXPSL 39 * Regularization 5-7 Penalty : on FC Layer
SQBUJBMTNPPUIOFTTQFOBMUZ: on output feature : ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) 5-7 Penalty spatial smoothness penalty Combined Feature Pixel shuffle : Loss Function For Scrambled Image Classification = => + A 5-7 + B B

1SPQPTFE"EBQUBUJPO/FUXPSL 40 ・5-7 Penalty - Sum in each row and
line is 1 5-7 = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 + 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99 ＝ 1 1 ＝

1SPQPTFE"EBQUBUJPO/FUXPSL 41 ・SQBUJBMTNPPUIOFTTQFOBMUZ ・・・ spatial smoothness penalty Combined Feature Pixel
shuffle Usual natural image Combined feature value position value position Combined feature is zig-zaged :

1SPQPTFE"EBQUBUJPO/FUXPSL 42 ・SQBUJBMTNPPUIOFTTQFOBMUZ Usual natural image Combined feature value position
value position Combined feature is zig-zaged ↓ Minimize gap between Adjacent pixels (Spatial smoothness penalty, B ) B = 1 E FG5 P [B,F R + B,F (S)]

1SPQPTFE"EBQUBUJPO/FUXPSL 43 ・SQBUJBMTNPPUIOFTTQFOBMUZ .JOJNJ[FBEKBDFOUQJYFMHBQ B,F R = 1 −
1 E XG5 R E YG5 Z-5 E [G5 = [: F R (ℎ, , ) ]7 B,F S = 1 − 1 E XG5 R-5 E YG5 Z E [G5 = [: F S (ℎ, , ) ]7 17 27 35 23 15 25 30 21 20 30 43 22 20 30 43 22 Horizontal Difference( B,F (R)) Vertical difference ( B,F (S)) :

,FZ4QBDF0O-& 44 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y Y VQQFSCJUBOEMPXFSCJU 1JYFMTIVGGMF _B
7 a 6 ! /FHBUJWF1PTJUJWF5SBOTGPSNBUJPO d_ 2fgah = _B ɾd_ = 7 a 6 ! a 2fgah

,FZ4QBDF0O&U$ 45 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y 3PUBUJPO*OWFSTJPO k&F (4 a 2)H
/FHBUJWF1PTJUJWF5SBOTGPSNBUJPO d_ 2H $PMPSDPNQPOFOUTIVGGMJOH [no 6H #MPDL-PDBUJPO4IVGGMJOH pB ! = k&F ɾd_ a [no a pB = 8H a 2H a 6H a !

,FZ4QBDF0O&-& 46 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y %JGGFSFOULFZ4IVGGMJOH s_B { 7
a 6 ! a 2fgah}H #MPDL-PDBUJPO4IVGGMJOH pB ! = s_B ɾpB = { 7 a 6 ! a 2fgah}Ha !

1SPQPTFE.FUIPE 47 ・5-7 Penalty - Sum in each row and
line is 1 5-7 = knY + oFdv ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99 ＝ 1 1 ＝

1SPQPTFE.FUIPE 48 ・5-7 Penalty - Sum in each row is
1 knY = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99

1SPQPTFE.FUIPE 49 ・5-7 Penalty - Sum in each line is
1 oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99

1SPQPTFE.FUIPE 50 8IZ5-7 Penalty works - (ex) Sum in each
line oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 F,I : L1 regularization (sparseness constraint) -> remain few important parameter and others to be 0 F,I 7 : L2 regularization (minimize parameter) -> minimize unimportant parameter to be 0

1SPQPTFE.FUIPE 51 8IZ5-7 Penalty works - (ex) Sum in each
line oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 Red part : converge if only one F,I is 1 and other F,I is zero (ex) (0,0,0,1,0,0,0)

AAAIWS2020_oral_pptx

AAAIWS2020_oral_pptx

More Decks by koki madono

Other Decks in Research

Featured

Transcript