Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Aptos2019 48th Solution

icebee16
September 23, 2019

Aptos2019 48th Solution

icebee16

September 23, 2019
Tweet

More Decks by icebee16

Other Decks in Technology

Transcript

  1. @icebee__ @mocobt
    UI4PMVUJPOίϯϖͷ;Γ͔͑Γ
    ࣗ࡞ύΠϓϥΠϯͱ͋Ε͜Ε
    "1504#MJOEOFTT%FUFDUJPO

    View Slide

  2. ࣗݾ঺հ
    @icebee__
    •ۚ༥ܥͷݚڀһ
    •ࠓճͷߩݙ͸ɼύΠϓϥΠϯͷ࡞੒ʹΑΔ࣮ݧͷޮ཰Խ
    •CGͷR&D
    •ࠓճͷߩݙ͸ɼจݙௐࠪ΍EDAʹΑΔfinding
    @mocobt
    Kaggleʹ͸νʔϜݻఆͰࢀՃ͍ͯ͠·͢ʂʂ
    ࢀߟ: https://mocobt.hatenablog.com/entry/2019/07/14/013922

    View Slide

  3. 1ϖʔδͰΘ͔Δ APTOS 2019
    Input: ؟ఈը૾ Output: ओ؍ධՁ஋
    ݈߁ ةݥ
    0 1 2 3 4
    ౶೘ප໢ບ঱ͷॏ঱౓෼ྨίϯϖςΟγϣϯ
    ࢀߟ: https://www.kaggle.com/c/aptos2019-blindness-detection
    https://en.wikipedia.org/wiki/Ophthalmoscopy

    View Slide

  4. APTOSͷ݁Ռ
    …ͳΜͰνʔϜ໊͕ʮΈ͔Μʯͳͷʁ
    Private 4IBLF͠ͱΔʙ
    4IBLF͠ͱΔʙ
    (Top 2%)
    Public ͠Ί͖Γ౰೔ʹΑ͏΍͘
    (Top 8%)

    View Slide

  5. Έ͔Μͷ༝དྷ
    ※ը૾͸ΠϝʔδͰ͢ɽҎԼɼϓϥΠόγʔΛߟྀͯ͠ɼΈ͔Μͷը૾Λ༻͍·͢ɽ
    ؟ఈը૾͕Έ͔Μʹݟ͖͑ͯͨɽ
    ඒຯͦ͠͏… Έ͔Μ৯΂͍ͨ…
    αΠί͔ͳʁ (໧ͬͯνʔϜ໊ʹઃఆ)

    View Slide

  6. Agenda
    ࣮ݧޮ཰Խ
    Our Solution

    View Slide

  7. લ൒: Our Solution
    ࣮ݧޮ཰Խ
    Our Solution
    Our Solution

    View Slide

  8. Problems & Approaches
    ෼ྨͰղ͔͘ʁճؼͰղ͍ͨޙʹᮢ஋ॲཧ͢Δ͔ʁ
    ճؼͰղ͍ͨޙʹɼᮢ஋࠷దԽ
    Ξϯαϯϒϧͷ΍Γํ͕Ṗ
    ճؼϞσϧͷग़ྗฏۉ + ᮢ஋΋ฏۉԽ
    ϥϕϧ͕͋Γ͑Μ΄Ͳෆۉߧ
    ಉλεΫ޲͚σʔλͰਫ૿͠
    ը૾αΠζ΍ً౓͕όϥόϥ
    Original CropͰରԠ
    Our Solution

    View Slide

  9. ճؼͰղ͍ͨޙʹɼᮢ஋࠷దԽ
    ݁Ռతʹɼճؼ+ᮢ஋ॲཧ͕ڧ͔ͬͨɽ
    ᮢ஋ͷ࠷దԽख๏: https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa
    ෼ྨ (Classification)
    ճؼ (Regression) + ᮢ஋ॲཧ
    Our Solution: ෼ྨͰղ͔͘ʁճؼͰղ͍ͨޙʹᮢ஋ॲཧ͢Δ͔ʁ
    Class
    ֬཰
    0 1 2 3
    4
    0.1 0.4 0.3 0.15 0.05
    Class 1
    0.0
    0 1 2 3
    Value
    Class
    0.8 1.2 2.6 3.2 4.0
    ग़ྗ: 1.4
    Class 2
    4

    View Slide

  10. ճؼϞσϧͷग़ྗฏۉ + ᮢ஋΋ฏۉԽ
    Our Solution: Ξϯαϯϒϧͷ΍Γํ͕Ṗ
    ֤ϞσϧຖʹInference + શϞσϧͷฏۉᮢ஋Ͱ෼ྨ

    Ϟσϧ1ͷ༧ଌ஋
    Ϟσϧ2ͷ༧ଌ஋
    Ϟσϧnͷ༧ଌ஋
    1.54
    2.16
    1.87
    }ฏۉ༧ଌ஋
    1.93

    ᮢ஋1 ᮢ஋2 ᮢ஋3 ᮢ஋4
    ộ ộ ộ
    Ϟσϧ1ͷᮢ஋ 0.90 1.62 2.64 3.31
    Ϟσϧ2ͷᮢ஋ 0.82 1.53 2.36 3.39

    Ϟσϧnͷᮢ஋ 0.76 1.77 2.75 3.46
    ฏۉᮢ஋ 0.85 1.89 2.55 3.33
    Class 2
    ֤ϞσϧͰClassΛ֬ఆ͔ͯ͠ΒฏۉऔΔλΠϓͷΞϯαϯϒϧ͸৳ͼͳ͔ͬͨ

    View Slide

  11. Traditional EDA of APTOS Data
    • Class 0 ͕൒਺Ҏ্
    • Class 4 ͕ѹ౗తʹগͳ͍
    Our Solution: ϥϕϧ͕͋Γ͑Μ΄Ͳෆۉߧ

    View Slide

  12. ಉλεΫ޲͚σʔλͰਫ૿͠
    2015೥։࠵ͷྨࣅίϯϖDataΛҰ෦ར༻ (ҎԼɼDRD 2015)
    ࢀߟ: https://www.kaggle.com/c/diabetic-retinopathy-detection
    Resizeࡁ: https://www.kaggle.com/benjaminwarner/resized-2015-2019-blindness-detection-images
    ࢖͍ํ
    ύλʔϯ1: DRD 2015ͷClass 0Ҏ֎Λશͯ࢖༻
    ύλʔϯ2: APTOSͷ֤Classͷׂ߹Λอͬͨ··ɼՄೳͳݶΓDRD 2015Ͱਫ૿͠
    ͦͷଞTraining Dataʹର͢ΔରԠ
    APTOS͔Βॏෳը૾Λ͢΂ͯআڈ
    ࢀߟ: https://www.kaggle.com/maxwell110/duplicated-list-csv-file
    Maxwell͞Μ͋Γ͕ͱ͏͍͟͝·͢
    Our Solution: ϥϕϧ͕͋Γ͑Μ΄Ͳෆۉߧ

    View Slide

  13. Traditional EDA of APTOS Data
    • Class 0 ͕൒਺Ҏ্
    • Class 4 ͕ѹ౗తʹগͳ͍
    • ը૾αΠζ΋όϥόϥ…
    • ϦʔΫͯ͠Δ…ʁ
    • 480x640ͷը૾ଟ͗͢
    Our Solution: ը૾αΠζ΍ً౓͕όϥόϥ

    View Slide

  14. Step 1 (in) Step 2 Step 3 Step 4 Step 5 (out)
    Ben’s CropͰً౓౷Ұ
    (1st [email protected] 2015)
    Original CropͰରԠ
    Our Solution: ը૾αΠζ΍ً౓͕όϥόϥ
    • த৺ͱΤοδ͔Β൒ܘऔಘ
    • தԝʹը૾഑ஔ&ϦαΠζ
    APTOS ؟ఈը૾ͷ໰୊఺
    ܗঢ়͕҆ఆ͠ͳ͍ (ԁܗ or ΍͚ʹ֦େ͞Ε͍ͯΔ)
    ը૾αΠζͱً౓͕όϥόϥ
    Input: ؟ఈը૾ Output: ը૾αΠζ, ً౓, (͋Δఔ౓)ܗঢ়͕Ұఆͷը૾
    Assumption: ؟ఈ͕΄΅ԁܗ
    Pros: όϥόϥ໰୊Λ͍͍ͩͨղܾ Cons: ͪΐͬͱ஗͍
    Ben’s Edge Crop
    • ෳ਺ճɼ௚ઢऔಘ
    • ަ఺͔Βԁத৺Λܭࢉ
    ԁ্ೋ఺Λ݁Ϳઢ෼͔Βɼ
    ԁͷத৺Λ௨Δ௚ઢΛಋग़
    Τοδݕग़
    (Canny๏)

    View Slide

  15. Our Total Solution
    Mean
    Average
    Blending
    Private: 48th / 2943
    LB: 0.9259 (Top 2%)
    Public: 230th / 2943
    LB: 0.8078 (Top 8%)
    Link: https://www.kaggle.com/muhakabartay/bump-to-0-800-e-2ln-more-efficient
    Most stable Kernel
    DRD 2015
    APTOS
    (duplicated removed)
    Remove 0 label
    Preserve the label ratio of 2019
    Train
    Train
    Preprocessing
    Preprocessing
    Ben’s Edge Crop
    (Size: 320)
    Normalization with ImageNet stats
    Normalization with ImageNet stats
    Train
    Training (regression)
    • EfficientNet-b4 (pretrained)
    • Batch size: 16
    • Adam (lr: 0.001, weight_decay: 1e-5)
    • MSELoss
    • Augmentation
    • RandomRotation
    • Random HorizontalFlip
    • Threshold optimization
    • EfficientNet-b4 (pretrained)
    • Batch size: 32
    • Adam (lr: 0.001, weight_decay: 1e-5)
    • MSELoss
    • Augmentation
    • RandomRotation
    • Random HorizontalFlip
    • Threshold optimization
    Training (regression)
    Training (regression)
    • EfficientNet-b5 (pretrained)
    • Batchsize: 64
    • Adam (lr: 0.001, weight_decay: 1e-5)
    • QWK (early stopping: MSELoss)
    • Augmentation:
    • Defaults of fast.ai
    • Random VerticalFlip
    • Threshold optimization
    Preprocessing
    [fast.ai’s transform]
    Resize: squish
    Padding: reflection
    Only Resizing
    (Size: 256)
    Stratified 1fold
    Validation
    Stratified 5fold
    Validation
    Random split
    Validation
    Our Solution
    Prediction
    Prediction
    Prediction
    • Fold: 5
    • TTA: 5
    • RandomRotate
    • RandomHorizontalFlip
    • Fold: 1
    • TTA: 10
    • RandomRotate
    • RandomHorizontalFlip
    • Fold: 1
    • TTA: None
    • Weight: 5
    Private LB: 0.923899
    Public LB: 0.806519
    Private LB: 0.917967
    Public LB: 0.790956
    Private LB: 0.910…
    Public LB: 0.792…
    25
    10
    5
    @icebee__
    @mocobt
    Ben’s Edge Crop
    (Size: 256)

    View Slide

  16. ࢼͯ͠μϝͩͬͨ͜ͱ
    DRD2015ͰPre-training → APTOSͰ Training
    Validation͕ѱ͔ͬͨͷ͔ɼ৳ͼͣ
    Messidor Databaseͷར༻
    ϥϕϧఆ͕ٛҧ͏͍͔ͤɼ৳ͼͣ
    Class 0Λ൑ผ͢ΔϞσϧ࡞੒ →ผϞσϧͷ༧ଌʹ࢖༻
    Public͸Class 1, 2, 3ΛClass 0ͱޡ൑அ͢Δ͜ͱ͕ଟ͘ɼ͋·Γޮ͔ͣ
    ঘɼPrivateͰ͸ޮ͍͍ͯͨ໛༷
    Weighted MSE LossΛద༻ (਺͕গͳ͍Classʹ܏ࣼ)
    ΄΅৳ͼͣ
    Our Solution

    View Slide

  17. ࢼͤͳ͔ͬͨ͜ͱ
    Our Solution
    Batch size > 32 or Image size > 320
    Memory଍Γͣɽͨͩfast.aiͰ͸Batch size 64͕P100Ͱී௨ʹಈ͍͍ͯͨ…
    ্Ґ੎͸ը૾αΠζ͕େ͖͔ͬͨɽ࣮૷͕ແཧͩͱఘΊ͍ͯͨͷͰɼ൓ল…
    Inception, SEResNeXt, DenseNet165ͳͲଞϞσϧ
    EfficientNetͰखҰഋͩͬͨ…(Dense͸ࢼ͕ͨ͠ɼฃΘͳ͔ͬͨ)
    GradCAMͳͲͷՄࢹԽख๏Λ༻͍ͨEDA
    ࣮૷ؒʹ߹Θͣ… ͜Ε͸ຊ౰ʹࢼͨ͠΄͏͕ྑ͔ͬͨ… (ࢀߟ: Nakama>ω ͞Μͷϒϩά)
    Pseudo labeling
    ๨Εͯͨ

    View Slide

  18. ޙ൒: ࣮ݧޮ཰Խ
    ࣮ݧޮ཰Խ
    Our Solution
    ࣮ݧޮ཰Խ

    View Slide

  19. ࣮ݧͷޮ཰Խ
    ࣮ݧޮ཰Խ
    ύΠϓϥΠϯͰࣗಈԽ ࣮ݧ؅ཧ

    View Slide

  20. ύΠϓϥΠϯུ֓ਤ
    ࣮ݧޮ཰Խ: ύΠϓϥΠϯͰࣗಈԽ
    Preprocessing Training Prediction

    View Slide

  21. ύΠϓϥΠϯུ֓ਤ
    Preprocessing Training Prediction
    ࣮ݧޮ཰Խ: ύΠϓϥΠϯͰࣗಈԽ

    View Slide

  22. ύΠϓϥΠϯུ֓ਤ
    Preprocessing Training Prediction
    Config
    Train
    ✓Save as caches
    Preprocessed
    Log
    Model
    ✓Main log
    ✓Learning curve
    ✓Confusion Matrix
    ✓Pytorch Model
    ✓Threshold data
    Config
    Preprocessed
    Test
    Config
    Model
    Submit
    ࣮ݧޮ཰Խ: ύΠϓϥΠϯͰࣗಈԽ

    View Slide

  23. ύΠϓϥΠϯ·ͱΊ
    Kernel࢖༻ن੍໰୊΋ɼTraining͸΄΅Өڹͳ͠
    ίϯϖޙظ͸GCPͰΠϯελϯεΛཱͯ·ͬͯͨ͘
    Submit࣌ͷʮKernel্ཱ͕ͪΒͶ͑…ͳΜ΍͜Ε…ʯ໰୊͸ྲྀੴʹରॲͰ͖ͣ
    ύΠϓϥΠϯ͔ΒAPIܦ༝ͰDatasetΛUploadͯ͠ɼखؒ࡟ݮ
    Kaggle APIͷDocument͕ෆ଍͍ͯͯ͠ɼͦͦۤ͜͜࿑ͨ͠
    धཁ͕͋ΔͳΒɼ·ͱΊͯهࣄʹ͠·͢
    ͨͩ͠ɼύΠϓϥΠϯपΓ͸ڽΓ͗͢ΔͱίέΔ (ܦݧஊ)
    ࣮૷ίετ͹͔Γ͔͞Έɼຊےͷίϯϖ΁ͷऔΓ૊Έ͕ૄ͔ʹͳΓ͕ͪ
    ࣮ݧޮ཰Խ: ύΠϓϥΠϯͰࣗಈԽ

    View Slide

  24. SlackʹΑΔ࣮ݧ؅ཧ
    ࣮ݧޮ཰Խ: ࣮ݧ؅ཧ
    …ͳΔ΄Ͳ..Α͘Θ͔Βͳ͍…
    Log༻ͷChannel΋࡞͕ͬͨɼฤू͠ʹ͍͘ & աڈͷ݁ՌΛৼΓฦΓʹ͔ͬͨ͘

    View Slide

  25. ࠷ڧͷ࣮ݧ؅ཧ
    ~ Grand Excel Master ~
    ࣮ݧID: Kaggle Datasetͷ؅ཧ༻ LB & CV Ծઆ & ࣮૷಺༰ & ݁Ռ
    ࣮ݧޮ཰Խ: ࣮ݧ؅ཧ

    View Slide

  26. ࣮ݧID: Kaggle Datasetͷ؅ཧ༻ LB & CV Ծઆ & ࣮૷಺༰ & ݁Ռ
    ࣮ݧ؅ཧ͔ΒΘ͔Δ൓ল఺
    Ծઆͷཱͯํ͕Ṗ
    • ଞॴͰݟͨ஌ݟʹج͍ͮͨԾઆ͚ͩ
    • EDAʹج͍ͮͨԾઆΛཱͯΒΕ͍ͯͳ͍
    ۭཝଟ͠
    • ݁ՌΛ൓ᢸ͍ͯ͠ͳ͍
    • ͦ΋ͦ΋ࠞಉߦྻΛݟͨͷ͕ʒ5೔લͱ͔
    • ·͋…औΕͨͷͰ…
    ࣮ݧޮ཰Խ: ࣮ݧ؅ཧ

    View Slide

  27. ࠓճɼ࣮ࡍʹ͔͔͓ͬͨۚ
    ࣮ݧޮ཰Խ
    35,214ԁ
    ΪοΫϦࠊ࣏ྍඅ
    (ίϯϖऴྃ௚ޙʹෛই)
    53,030ԁ

    View Slide

  28. ·ͱΊ
    APTOSͰԿͱ͔ۜϝμϧήοτ
    Original CropͰ্ͦͦ͜͜Ґߦ͚ͨ
    ը૾αΠζͱPseudo Labeling্͕Ґ੎ͱͷେ͖ͳࠩͩͬͨ
    ΞʔΩςΫνϟͷҧ͍͸ͦ͜·Ͱॏཁͳ໰୊Ͱ͸ͳ͍…ͱࢥ͏
    ࣮ݧޮ཰Խʹ͸ύΠϓϥΠϯศརɼKaggleAPI΋ศར
    (չ͍͕͠) Excel͸݁ՌͷৼΓฦΓָ͕
    ΪοΫϦࠊ͸೑ମతʹ΋ۚમతʹ΋ͱͯ΋ͭΒ͍
    (͓·͚) RepositoryͱৼΓฦΓϒϩά
    • https://github.com/icebee16/kaggle_APTOS2019
    • http://icebee.hatenablog.com/entry/2019/09/10/221351
    • https://mocobt.hatenablog.com/entry/2019/09/09/013658

    View Slide

  29. References
    ίϯϖͱಉ͡λεΫΛѻͬͨ࿦จ
    [Sayres et al. Ophthalmology 2019, Volume 126, Issue 4, Pages 552-564]
    [Krause et al. Ophthalmology 2018, Volume 125, Issue 8, Pages 1264-1272]
    [Poplin et al. Nature Biomedical Engineering 2018]
    [Gulshan et al. JAMA, The Journal of American Medical Association 2016, 316(22)]
    Survey Papers
    [Qureshi et al. Symmetry 2019, 11(6), 749]
    [Erfuth et al. Progress in Retinal and Eye Research 2018, Volume 67, Pages 1-29]
    [Fenner et al. Ophthalmology and Therapy 2018, Volume 7, Issue 2, Pages 333-346]
    [Almotiri et al. Applied Sciences 2018, 8(2), 155]

    View Slide