pixiv App Night 20190611

6月11日に開催された「pixiv App Night」の発表資料です。

「わたしは機械学習プロジェクトで
技術的負債を抱えました」

Transcript

1. C-LIS CO., LTD.
2. QJYJW
   "QQ
   /JHIU
    Θͨ͠͸ػցֶशϓϩδΣΫτͰ
 ٕज़తෛ࠴Λ๊͑·ͨ͠
   5FOTPS'MPXͰझຯͷը૾ऩूαʔόʔΛ࡞Δɹ೥݄߸
3. C-LIS CO., LTD.
    ༗ࢁ
   ܓೋ
   ʢ,FJKJ
   "3*:"."ʣ $-*4
   $0
   -5% Photo : Koji MORIGUCHI (MORIGCHOWDER) "OESPJEΞϓϦ։ൃνϣοτσΩϧ ʮझຯͰػցֶशΛ΍͍ͬͯΔऀͩʯ ΍ͬͯ·ͤΜ
4. 5FOTPS'MPXൃදʢ೥݄ʣ
   
5. ػցֶशΛར༻ͨ͋͠Ε͜Ε
    IUUQLJWBOUJVNIBUFCMPKQFOUSZ TensorFlowͰΞχϝΏΔΏΓͷ੍࡞ձࣾΛࣝผ͢Δ IUUQCPIFNJBIBUFOBCMPHDPNFOUSZ σΟʔϓϥʔχϯάͰ͓ͦদ͞Μͷ࿡ͭࢠ͸ݟ෼͚ΒΕΔͷ͔ʁ IUUQCPIFNJBIBUFOBCMPHDPNFOUSZ IUUQDISJTUJOBIBUFOBCMPHDPNFOUSZ Deep LearningͰϥϒϥΠϒʂΩϟϥΛࣝผ͢Δ
6. 
    ؟ڸ່ͬͷΠϥετΛ
 Πϯλʔωοτ͔ΒࣗಈͰऩू͍ͨ͠ʂ
7. ؟ڸ່ͬ൑ఆ
   
8. ؟ ڸ ͬ ່ ˜
   ࠜઇΕ͍
9. 
   
10. 
     σϞ
11. 
     ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ ෇͚ σʔληοτ؅ཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ෇͚ σʔληοτ
 ؅ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧ഑ஔ ը૾ૹ৴ ൑ఆ݁Ռ
12. ϥϕϧͷछྨ PSJHJOBM@BSU
    OTGX
    GBWPSJUF
    QIPUP
    JMMVTU
    DPNJD
     GBDF
    GFNBMF
    NFHBOF
    TDISPPM@VOJGPSN
    CMB[FS@VOJGPSN
    TBJMPS@VOJGPSN HM
    LFNPOP
    NBMF
    CM
    DBU
    EPH GPPE
    EJTMJLF
13. https://twitter.com/_meganeco .FHBOF$P
    1MBZHSPVOE
     https://playground.megane.ai/
14. 
     ̏೥ؒͷࣦഊ
15. ֶशϓϩηεʹOPIVQΛ͚ͭ๨ΕΔ
     $ nohup python train.py ...
16. ϗʔϜσΟϨΫτϦΛਧ͖ඈ͹͢
     $ rsync -av --delete ./scripts/ meganeco2.c-lis.net:~/
17. αʔόʔηΩϡϦςΟΛమนʹ
     $ iptables -F IUUQXXXJOWJUSPKQCMPHJOEFYDHJ-JOVY@IUN
18. def download(self, url, output_dir): response = requests.get(url) response.raise_for_status() fd, file_path = tempfile.mkstemp(dir=output_dir) os.close(fd) with open(file_path, mode='wb') as fp: fp.write(response.content) response.close() return file_path
     ϑΝΠϧσΟεΫϦϓλΛރׇͤ͞Δ IUUQTOJTIJNVSBIBUFOBEJBSZPSHFOUSZ
19. 
     ੵΈ্͛ΒΕٕͨज़తෛ࠴ͷ਺ʑ
20. ٕज़తෛ࠴ ϞμϯͰͳ͍"OESPJEΞϓϦͷίʔυ
    5FOTPS'MPX
    ରԠ
    ֶशͱධՁϓϩηεͷ෼཭
    ;͘Ε͕͋Δσʔληοτ
    
    
21. 
     ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ ෇͚ σʔληοτ؅ཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ෇͚ σʔληοτ
 ؅ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧ഑ஔ ը૾ૹ৴ ൑ఆ݁Ռ
22. ϞμϯͰͳ͍"OESPJEΞϓϦͷίʔυʢٕज़ʣ ,PUMJO
    3Y,PUMJO
    3PPN
    #VUUFS,OJGF
    5FOTPS'MPX
    -JUF
    
     $PSPVUJOFT "OESPJE
    "SDIJUFDUVSF
    $PNQPOFOUT
    /FVSBM
    /FUXPSLT
    "1*
23. 
     5FOTPS'MPX
    ରԠ
24. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From model_res5.py:28: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead.
    
25. 
     def layers(tag_name, image, keep_prob=0.5, training=False): scope = '%s/%s' % (tag_name, NAME) with tf.variable_scope(scope): conv = tf.layers.conv2d(image, BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, trainable=training, name='conv_top') def layers(tag_name, image, rate=0.5, training=False): scope = '%s/%s' % (tag_name, NAME) with tf.variable_scope(scope): conv = tf.keras.layers.Conv2D(BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, name='conv_top')(image)
26. 
     IUUQTUXJUUFSDPNLFJKJ@BSJZBNBTUBUVT
27. 
     ֶशͱධՁϓϩηεͷ෼཭
28. 
     ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ ෇͚ σʔληοτ؅ཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ෇͚ σʔληοτ
 ؅ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧ഑ஔ ը૾ૹ৴ ൑ఆ݁Ռ
29. ߴՐྗίϯϐϡʔςΟϯάαʔόʔʢݕূ࣮ݧػʣ $16ɹɹɹɹ9FPO
    &
    W
    ίΞ
    ʷ
    .FNPSZɹɹ(#
    ετϨʔδɹ
    44%
    (#
    ʷʢ3"*%ʣ
    (16ɹɹɹ
    
    
    /7*%*"
    5*5"/
    9
    (#
    ɹɹɹɹɹ
    
    
    
    /7*%*"
    5*5"/
    9
    (#
    ɹɹɹɹɹ
    
    
    
    /7*%*"
    (F'PSDF
    (59
    5J
    (#
    ɹɹɹɹ
    
    
    ɹ
    /7*%*"
    (F'PSDF
    (59
    5J
    (#
     ʢDriver Version: 410.48ʣ
30. ߴՐྗίϯϐϡʔςΟϯάαʔόʔʢݕূ࣮ݧػʣ $16ɹɹɹɹ9FPO
    &
    W
    ίΞ
    ʷ
    .FNPSZɹɹ(#
    ετϨʔδɹ
    44%
    (#
    ʷʢ3"*%ʣ
    (16ɹɹɹ
    
    
    ".%
    3BEFPO
    7FHB
    (#
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    ".%
    3BEFPO
    7FHB
    (#
    
31. 
    
32. ̎ͭͷϓϩηεʢֶशͱධՁʣ
     train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛ؂ࢹ ߋ৽
33. - name: train-single-rocm container: image: rocm/tensorflow:rocm2.2-tf1.13-python3 command: [/workdir/scripts/train_single.sh] env: - name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: amd.com/gpu: 2 requests: amd.com/gpu: 2
     - name: train-single-cuda container: image: tensorflow/tensorflow:1.13.1-gpu-py3 command: [/workdir/scripts/train_single.sh] env: - name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: nvidia.com/gpu: 2 requests: nvidia.com/gpu: 2
34. #!/bin/sh HIP_VISIBLE_DEVICES=0 CUDA_VISIBLE_DEVICES=0 \ python3 sources/server/picture_single_discriminator/eval.py \ --learning_config $CONFIG_FILE \ --tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR & HIP_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1 \ python3 sources/server/picture_single_discriminator/train.py \ --learning_config $CONFIG_FILE \ --tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR \ --batch_size $BATCH_SIZE \ --learning_rate $LERANING_RATE \ --num_gpus 1 \ --step $STEPS
    
35. ධՁϓϩηεΛഇࢭ
     train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛ؂ࢹ ߋ৽ ×
36. ֶशɾධՁϓϩηε
     train & eval process ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽
37. 
     ;͘Ε͕͋Δσʔληοτ
38. 
     ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ ෇͚ σʔληοτ؅ཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ෇͚ σʔληοτ
 ؅ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧ഑ஔ ը૾ૹ৴ ൑ఆ݁Ռ
39. σʔληοτ؅ཧαʔόʔ ໿ສຕͷը૾ϑΝΠϧʢ໿5#ʣ
    ໿ສ݅ͷϥϕϧ
    όοΫΞοϓମ੍ͷ੔උ
    
    
40. 
    
41. ٕज़తෛ࠴Λͳͥղܾ͠ͳ͍ͷ͔
     ͍ͭ·Ͱ΍ΔʁɹͲ͜·Ͱ΍Δʁɹকདྷͷల๬͸ʁ ҆ఆͯ͠Քಇ͍ͯ͠ΔγεςϜΛյͨ͘͠ͳ͍
 ʢJG
    JU
    XPSLT
    EPOU
    pY
    JUʣ όοΫΞοϓϓϥϯΛ༻ҙͰ͖ͳ͍ʢܦࡁ߹ཧੑʣ
42. 
     ͍ͭ·Ͱ΍ΔʁɹͲ͜·Ͱ΍Δʁɹকདྷͷల๬͸ʁ
43. C-LIS CO., LTD. ຊࢿྉ͸ɺ༗ݶձࣾγʔϦεͷஶ࡞෺Ͱ͢ɻຊࢿྉͷશ෦ɺ·ͨ͸Ұ෦ʹ͍ͭͯɺஶ࡞ऀ͔ΒจॻʹΑΔڐ୚Λಘͣʹෳ੡͢Δ͜ͱ͸ې͡ΒΕ͍ͯ·͢ɻ 5IF
    "OESPJE
    4UVEJP
    JDPO
    JT
    SFQSPEVDFE
    PS
    NPEJpFE
    GSPN
    XPSL
    DSFBUFE
    BOE
    TIBSFE
    CZ
    (PPHMF
    BOE
    VTFE
    BDDPSEJOH
    UP
    UFSNT
    EFTDSJCFE
    JO
    UIF
    $SFBUJWF
    $PNNPOT
    
    "UUSJCVUJPO
    -JDFOTF ֤੡඼໊ɾϒϥϯυ໊ɺձ໊ࣾͳͲ͸ɺҰൠʹ֤ࣾͷ঎ඪ·ͨ͸ొ࿥঎ඪͰ͢ɻຊࢿྉதͰ͸ɺ˜ɺšɺäΛׂѪ͍ͯ͠·͢ɻ 5IF
    "OESPJE
    SPCPU
    JT
    SFQSPEVDFE
    PS
    NPEJpFE
    GSPN
    XPSL
    DSFBUFE
    BOE
    TIBSFE
    CZ
    (PPHMF
    BOE
    VTFE
    BDDPSEJOH
    UP
    UFSNT
    EFTDSJCFE
    JO
    UIF
    $SFBUJWF
    $PNNPOT
    
    "UUSJCVUJPO
    -JDFOTF