Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AAAIWS2020_oral_pptx

koki madono
February 07, 2020

 AAAIWS2020_oral_pptx

koki madono

February 07, 2020
Tweet

More Decks by koki madono

Other Decks in Research

Transcript

  1. %FFQ-FBSOJOHXJUIPXOEBUB Use Deep Learning on public area 2 camera smartphone

    Take a photo In my own Share photos To the Human Human
  2. %FFQ-FBSOJOHXJUIPXOEBUB Use Deep Learning on public area 3 Human Send

    own photos Own face Pets Cloud server Classifier Deep learning
  3. %FFQ-FBSOJOHXJUIPXOEBUB Is this privacy preserved pipeline?? 4 Human Send own

    photos Own face Pets Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning
  4. %FFQ-FBSOJOHXJUIPXOEBUB Image Pixel shuffling : human non-understandable 9 Human Send

    own photos Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning
  5. %FFQ-FBSOJOHXJUIPXOEBUB Image Pixel shuffling : human non-understandable 10 Human Send

    own photos Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning Scrambled Image
  6. %FFQ-FBSOJOHXJUIPXOEBUB Adaptation Net: machine understandable 11 Human Send own photos

    Own face Pets Image Pixel shuffle Third party xxxx is used for training! Cloud server <worst case> Classifier Deep learning Adapt. Net Scrambled Image
  7. 4DSBNCMF*NBHF$MBTTJGJDBUJPO Goal 1. Training the classifier with high accuracy 2.

    Protecting the visual information (Privacy) of image - scramble : shuffle pixels / reverse pixel value 12 Original Image Scrambled Image (privacy protection) Image Scramble
  8. 3FMBUFE8PSL 5BOBLB 13 1 2 3 4 1 2 3

    4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1. shuffling 2 2 3 4 1 4 1 4 3 1 3 2 2 2 3 4 1 4 1 4 3 1 3 2 3 1 3 2 1 4 1 4 2 2 3 4 Rearrange intensity values In a row Rearrange intensity values As an RGB image 2. Reverse several values -1 -1 -1 -1 Block-wise operation B x B sized block input output Block-wise Image Scramble (same key in each block)
  9. 3FMBUFE8PSL 5BOBLB 14 1 2 3 4 1 2 3

    4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1. shuffling 2 2 3 4 1 4 1 4 3 1 3 2 2 2 3 4 1 4 1 4 3 1 3 2 3 1 3 2 1 4 1 4 2 2 3 4 Rearrange intensity values In a row Rearrange intensity values As an RGB image 2. Reverse several values -1 -1 -1 -1 Block-wise operation B x B sized block input output
  10. 3FMBUFE8PSL "EBQUBUJPO/FUXPSL 5BOBLB Goal - "EBQUBUJPOOFUXPSLFYUSBDUTGFBUVSFNBQGSPN TDSBNCMFEJNBHFT 15 3 1

    3 2 1 4 1 4 2 2 3 4 input Classifier CNN Adaptation Network one CNN layer Kernel : B x B (ex) ResNet VGG Block-Wise Feature Extraction
  11. $POUSJCVUJPOT 16 1. 1SPQPTF/FX*NBHF4DSBNCMFNFUIPE Object : Increasing the security level

    2. 1SPQPTF/FX"EBQUBUJPO/FUXPSL Object : Classify New Scrambling Image with high accuracy.
  12. 1SPQPTFE*NBHF4DSBNCMF.FUIPE 17 1. Extensive Learnable Encryption(ELE) - Consists of two

    scrambling methods Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration
  13. 1SPQPTFE*NBHF4DSBNCMF.FUIPE 18 1.1 Block Shuffling - Divide Block & shuffle

    location Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration
  14. 1SPQPTFE*NBHF4DSBNCMF.FUIPE 19 1.2 #MPDLXJTFTDSBNCMFPQFSBUJPO - Shuffle pixels with different keys

    in each blocks Image segmentation Block Shuffling Block-Wise Pixel Operation Block Integration
  15. 1SPQPTFE"EBQUBUJPO/FUXPSL 21 2.1 Block-wise Sub-network - support Block-Wise Pixel Operation

    Block-wise Sub-network Permutation Matrix Classifier Block-Wise Pixel Operation
  16. 1SPQPTFE"EBQUBUJPO/FUXPSL 22 2.2 Permutation Matrix - support Block Shuffling Block-wise

    Sub-network Permutation Matrix Classifier Block Shuffling
  17. 1SPQPTFE"EBQUBUJPO/FUXPSL 23 2.1 Block-wise Sub-network - : CNN (kernel size

    == block size) image segmentation (・, ) (・, ) ・・・ ・・・ (・, -) (・, ) ・・・ Input (32x32x3) blocks (4x4x3) x 64 feature (1x1x16) x 64
  18. 1SPQPTFE"EBQUBUJPO/FUXPSL 25 2.2 Permutation Matrix - Learning Block Shuffling by

    FC Layer ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (-) Output Input /
  19. &WBMVBUJPO4DSBNCMFE*NBHFT 27 Plain LE EtC ELE (proposed) Image Block-Wise key

    Same different different Block-Wise Pixel Shuffle ✔ ✔ Block Shuffling ✔ ✔
  20. 3FTVMU4FDVSJUZ&WBMVBUJPO 30 Plain LE EtC ELE (proposed) Image Key Space

    0 O(10566) O(10758) O(1055856) ELE(Proposed Scrambled Image)’s Key Space Become very large ↓ Increase the Security Level !
  21. 3FTVMU$MBTTJGJDBUJPO"DDVSBDZ 31 Dataset Adaptation Net Plain image LE EtC ELE

    (Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97%
  22. 3FTVMU$MBTTJGJDBUJPO"DDVSBDZ 32 Dataset Adaptation Net Plain image LE EtC ELE

    (Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97% Proposed Adaptation Network is not always effective (Plain Image & LE Image results)
  23. 3FTVMU$MBTTJGJDBUJPO"DDVSBDZ 33 Dataset Adaptation Net Plain image LE EtC ELE

    (Proposed) Cifar-10 No- Adaptation 96.70% 94.94% 85.94% 67.10% LE- Adaptation 95.64% 94.49% 80.16% 48.39% ELE- Adaptation 85.32% 87.28% 89.09% 83.03% Cifar-100 No- Adaptation 83.59% 78.25% 61.90% 43.05% LE- Adaptation 79.13% 75.48% 44.83% 7.19% ELE- Adaptation 60.36% 71.30% 71.91% 62.97% Proposed Adaptation Network is effective → Different key on blocks and Block Shuffling
  24. 4VNNBSZ 34 0CKFDU  *ODSFBTFTFDVSJUZPGTDSBNCMFEJNBHF  $MBTTJGZTDSBNCMFEJNBHFXJUIIJHIBDD 1SPQPTFE.FUIPE  &-&4DSBNCMFE*NBHF

     /FX"EBQUBUJPO/FUXPSL 3FTVMUT  *ODSFBTFTFDVSJUZ-FWFM  &YUSBDUGFBUVSFGSPN4DSBNCMFEJNBHF CZ"EBQUBUJPO/FUXPSL
  25. 1SPQPTFE"EBQUBUJPO/FUXPSL 36 2.2 Permutation Matrix Problem : Difficult to be

    a discrete value ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) input output
  26. 1SPQPTFE"EBQUBUJPO/FUXPSL 37 2.2 Permutation Matrix Problem : Difficult to be

    a discrete value Solution : Learning with Regularization ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) input output
  27. 1SPQPTFE"EBQUBUJPO/FUXPSL 38 * Regularization 5-7 Penalty : on FC Layer

    SQBUJBMTNPPUIOFTTQFOBMUZ: on output feature : ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) 5-7 Penalty spatial smoothness penalty Combined Feature Pixel shuffle :
  28. 1SPQPTFE"EBQUBUJPO/FUXPSL 39 * Regularization 5-7 Penalty : on FC Layer

    SQBUJBMTNPPUIOFTTQFOBMUZ: on output feature : ・・・ 1 0 0 0 ⋯ 0 0 0 1 ⋮ ⋱ ⋮ 0 1 0 0 ⋯ 0 0 1 0 ・・・ FC Layer (64 x 64) 5-7 Penalty spatial smoothness penalty Combined Feature Pixel shuffle : Loss Function For Scrambled Image Classification = => + A 5-7 + B B
  29. 1SPQPTFE"EBQUBUJPO/FUXPSL 40 ・5-7 Penalty - Sum in each row and

    line is 1 5-7 = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 + 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99 = 1 1 =
  30. 1SPQPTFE"EBQUBUJPO/FUXPSL 41 ・SQBUJBMTNPPUIOFTTQFOBMUZ ・・・ spatial smoothness penalty Combined Feature Pixel

    shuffle Usual natural image Combined feature value position value position Combined feature is zig-zaged :
  31. 1SPQPTFE"EBQUBUJPO/FUXPSL 42 ・SQBUJBMTNPPUIOFTTQFOBMUZ Usual natural image Combined feature value position

    value position Combined feature is zig-zaged ↓ Minimize gap between Adjacent pixels (Spatial smoothness penalty, B ) B = 1 E FG5 P [B,F R + B,F (S)]
  32. 1SPQPTFE"EBQUBUJPO/FUXPSL 43 ・SQBUJBMTNPPUIOFTTQFOBMUZ  .JOJNJ[FBEKBDFOUQJYFMHBQ B,F R = 1 −

    1 E XG5 R E YG5 Z-5 E [G5 = [: F R (ℎ, , ) ]7 B,F S = 1 − 1 E XG5 R-5 E YG5 Z E [G5 = [: F S (ℎ, , ) ]7 17 27 35 23 15 25 30 21 20 30 43 22 20 30 43 22 Horizontal Difference( B,F (R)) Vertical difference ( B,F (S)) :
  33. ,FZ4QBDF0O-& 44 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y Y  VQQFSCJUBOEMPXFSCJU 1JYFMTIVGGMF _B

     7 a 6 ! /FHBUJWF1PTJUJWF5SBOTGPSNBUJPO d_ 2fgah = _B ɾd_ = 7 a 6 ! a 2fgah
  34. ,FZ4QBDF0O&U$ 45 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y 3PUBUJPO*OWFSTJPO k&F (4 a 2)H

    /FHBUJWF1PTJUJWF5SBOTGPSNBUJPO d_ 2H $PMPSDPNQPOFOUTIVGGMJOH [no 6H #MPDL-PDBUJPO4IVGGMJOH pB ! = k&F ɾd_ a [no a pB = 8H a 2H a 6H a !
  35. ,FZ4QBDF0O&-& 46 //VNCFSPG#MPDLT ##MPDL4J[F *OQVU*NBHF)Y8Y %JGGFSFOULFZ4IVGGMJOH s_B  { 7

    a 6 ! a 2fgah}H #MPDL-PDBUJPO4IVGGMJOH pB ! = s_B ɾpB = { 7 a 6 ! a 2fgah}Ha !
  36. 1SPQPTFE.FUIPE 47 ・5-7 Penalty - Sum in each row and

    line is 1 5-7 = knY + oFdv ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99 = 1 1 =
  37. 1SPQPTFE.FUIPE 48 ・5-7 Penalty - Sum in each row is

    1 knY = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99
  38. 1SPQPTFE.FUIPE 49 ・5-7 Penalty - Sum in each line is

    1 oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 ・・・ 0.0 0.99 0.0 0.0 ⋯ 0.0 0.01 0.99 0.0 ⋮ ⋱ ⋮ 0.99 0.0 0.0 0.0 ⋯ 0.0 0.0 0.0 0.99
  39. 1SPQPTFE.FUIPE 50 8IZ5-7 Penalty works - (ex) Sum in each

    line oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 F,I : L1 regularization (sparseness constraint) -> remain few important parameter and others to be 0 F,I 7 : L2 regularization (minimize parameter) -> minimize unimportant parameter to be 0
  40. 1SPQPTFE.FUIPE 51 8IZ5-7 Penalty works - (ex) Sum in each

    line oFdv = 1 × E FG5 H E IG5 H F,I − (E IG5 H F,I 7 )5/7 Red part : converge if only one F,I is 1 and other F,I is zero (ex) (0,0,0,1,0,0,0)