pixiv App Night 20190611

Slide 1

Slide 1 text

C-LIS CO., LTD.

Slide 2

Slide 2 text

QJYJW"QQ/JHIU Θͨ͠͸ػցֶशϓϩδΣΫτͰ  ٕज़తෛ࠴Λ๊͑·ͨ͠ 5FOTPS'MPXͰझຯͷը૾ऩूαʔόʔΛ࡞Δɹ೥݄߸

Slide 3

Slide 3 text

C-LIS CO., LTD. ༗ࢁܓೋʢ,FJKJ"3*:"."ʣ $-*4$0 -5% Photo : Koji MORIGUCHI (MORIGCHOWDER) "OESPJEΞϓϦ։ൃνϣοτσΩϧ ʮझຯͰػցֶशΛ΍͍ͬͯΔऀͩʯ ΍ͬͯ·ͤΜ

Slide 4

Slide 4 text

5FOTPS'MPXൃදʢ೥݄ʣ

Slide 5

Slide 5 text

ػցֶशΛར༻ͨ͋͠Ε͜Ε IUUQLJWBOUJVNIBUFCMPKQFOUSZ TensorFlowͰΞχϝΏΔΏΓͷ੍࡞ձࣾΛࣝผ͢Δ IUUQCPIFNJBIBUFOBCMPHDPNFOUSZ σΟʔϓϥʔχϯάͰ͓ͦদ͞Μͷ࿡ͭࢠ͸ݟ෼͚ΒΕΔͷ͔ʁ IUUQCPIFNJBIBUFOBCMPHDPNFOUSZ IUUQDISJTUJOBIBUFOBCMPHDPNFOUSZ Deep LearningͰϥϒϥΠϒʂΩϟϥΛࣝผ͢Δ

Slide 6

Slide 6 text

؟ڸ່ͬͷΠϥετΛ  Πϯλʔωοτ͔ΒࣗಈͰऩू͍ͨ͠ʂ

Slide 7

Slide 7 text

؟ڸ່ͬ൑ఆ

Slide 8

Slide 8 text

؟ ڸ ͬ ່ ࠜઇΕ͍

Slide 9

Slide 9 text

Slide 10

Slide 10 text

σϞ

Slide 11

Slide 11 text

ධՁ༻αʔόʔ ܇࿅ɾֶश༻αʔόʔ܈ σʔληοτసૹ ʢTFRecordʣ ֶशࡁϞσϧऔಘ ը૾औಘ ը૾औಘ ϥϕϧ ෇͚ σʔληοτ؅ཧ αʔόʔ ը૾ऩूݩ ը૾ऩू ϥϕϧ ෇͚ σʔληοτ  ؅ཧΞϓϦ playground.megane.ai ֶशࡁΈϞσϧ഑ஔ ը૾ૹ৴ ൑ఆ݁Ռ

Slide 12

Slide 12 text

ϥϕϧͷछྨ PSJHJOBM@BSU OTGX GBWPSJUF QIPUP JMMVTU DPNJD GBDF GFNBMF NFHBOF TDISPPM@VOJGPSN CMB[FS@VOJGPSN TBJMPS@VOJGPSN HM LFNPOP NBMF CM DBU EPH GPPE EJTMJLF

Slide 13

Slide 13 text

https://twitter.com/_meganeco .FHBOF$P1MBZHSPVOE https://playground.megane.ai/

Slide 14

Slide 14 text

̏೥ؒͷࣦഊ

Slide 15

Slide 15 text

ֶशϓϩηεʹOPIVQΛ͚ͭ๨ΕΔ $ nohup python train.py ...

Slide 16

Slide 16 text

ϗʔϜσΟϨΫτϦΛਧ͖ඈ͹͢ $ rsync -av --delete ./scripts/ meganeco2.c-lis.net:~/

Slide 17

Slide 17 text

αʔόʔηΩϡϦςΟΛమนʹ $ iptables -F IUUQXXXJOWJUSPKQCMPHJOEFYDHJ-JOVY@IUN

Slide 18

Slide 18 text

def download(self, url, output_dir): response = requests.get(url) response.raise_for_status() fd, file_path = tempfile.mkstemp(dir=output_dir) os.close(fd) with open(file_path, mode='wb') as fp: fp.write(response.content) response.close() return file_path ϑΝΠϧσΟεΫϦϓλΛރׇͤ͞Δ IUUQTOJTIJNVSBIBUFOBEJBSZPSHFOUSZ

Slide 19

Slide 19 text

ੵΈ্͛ΒΕٕͨज़తෛ࠴ͷ਺ʑ

Slide 20

Slide 20 text

ٕज़తෛ࠴ ϞμϯͰͳ͍"OESPJEΞϓϦͷίʔυ 5FOTPS'MPXରԠ ֶशͱධՁϓϩηεͷ෼཭ ;͘Ε͕͋Δσʔληοτ

Slide 21

Slide 21 text

Slide 22

Slide 22 text

ϞμϯͰͳ͍"OESPJEΞϓϦͷίʔυʢٕज़ʣ ,PUMJO 3Y,PUMJO 3PPN #VUUFS,OJGF 5FOTPS'MPX-JUF $PSPVUJOFT "OESPJE"SDIJUFDUVSF$PNQPOFOUT /FVSBM/FUXPSLT"1*

Slide 23

Slide 23 text

5FOTPS'MPXରԠ

Slide 24

Slide 24 text

WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From model_res5.py:28: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead.

Slide 25

Slide 25 text

def layers(tag_name, image, keep_prob=0.5, training=False): scope = '%s/%s' % (tag_name, NAME) with tf.variable_scope(scope): conv = tf.layers.conv2d(image, BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, trainable=training, name='conv_top') def layers(tag_name, image, rate=0.5, training=False): scope = '%s/%s' % (tag_name, NAME) with tf.variable_scope(scope): conv = tf.keras.layers.Conv2D(BASE_CHANNEL, [1, 1], [1, 1], padding='SAME', activation=tf.nn.relu, use_bias=True, name='conv_top')(image)

Slide 26

Slide 26 text

IUUQTUXJUUFSDPNLFJKJ@BSJZBNBTUBUVT

Slide 27

Slide 27 text

ֶशͱධՁϓϩηεͷ෼཭

Slide 28

Slide 28 text

Slide 29

Slide 29 text

ߴՐྗίϯϐϡʔςΟϯάαʔόʔʢݕূ࣮ݧػʣ $16ɹɹɹɹ9FPO&WίΞʷ .FNPSZɹɹ(# ετϨʔδɹ44%(#ʷʢ3"*%ʣ (16ɹɹɹ/7*%*"5*5"/9(# ɹɹɹɹɹ/7*%*"5*5"/9(# ɹɹɹɹɹ/7*%*"(F'PSDF(595J(# ɹɹɹɹɹ/7*%*"(F'PSDF(595J(# ʢDriver Version: 410.48ʣ

Slide 30

Slide 30 text

ߴՐྗίϯϐϡʔςΟϯάαʔόʔʢݕূ࣮ݧػʣ $16ɹɹɹɹ9FPO&WίΞʷ .FNPSZɹɹ(# ετϨʔδɹ44%(#ʷʢ3"*%ʣ (16ɹɹɹ".%3BEFPO7FHB(# ".%3BEFPO7FHB(#

Slide 31

Slide 31 text

Slide 32

Slide 32 text

̎ͭͷϓϩηεʢֶशͱධՁʣ train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛ؂ࢹ ߋ৽

Slide 33

Slide 33 text

- name: train-single-rocm container: image: rocm/tensorflow:rocm2.2-tf1.13-python3 command: [/workdir/scripts/train_single.sh] env: - name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: amd.com/gpu: 2 requests: amd.com/gpu: 2 - name: train-single-cuda container: image: tensorflow/tensorflow:1.13.1-gpu-py3 command: [/workdir/scripts/train_single.sh] env: - name: DATASET_DIR value: "/dataset/tfrecords_classifier_latest" - name: CATALOGS_DIR value: "{{inputs.parameters.catalogs-dir}}" - name: TRAIN_DIR value: "data/{{inputs.parameters.train-dir}}" - name: SUMMARY_DIR value: "data/{{inputs.parameters.summary-dir}}" - name: CONFIG_FILE value: "scripts/{{inputs.parameters.config-file}}" - name: BATCH_SIZE value: "{{inputs.parameters.batch-size}}" - name: LERANING_RATE value: "{{inputs.parameters.learning-rate}}" - name: STEPS value: "{{inputs.parameters.steps}}" resources: limits: nvidia.com/gpu: 2 requests: nvidia.com/gpu: 2

Slide 34

Slide 34 text

#!/bin/sh HIP_VISIBLE_DEVICES=0 CUDA_VISIBLE_DEVICES=0 \ python3 sources/server/picture_single_discriminator/eval.py \ --learning_config $CONFIG_FILE \ --tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR & HIP_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1 \ python3 sources/server/picture_single_discriminator/train.py \ --learning_config $CONFIG_FILE \ --tfrecords_dir $DATASET_DIR \ --catalogs_dir $CATALOGS_DIR \ --train_dir $TRAIN_DIR \ --summary_dir $SUMMARY_DIR \ --batch_size $BATCH_SIZE \ --learning_rate $LERANING_RATE \ --num_gpus 1 \ --step $STEPS

Slide 35

Slide 35 text

ධՁϓϩηεΛഇࢭ train process eval process ֶशࡁΈύϥϝʔλʔ ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽ ධՁ มߋΛ؂ࢹ ߋ৽ ×

Slide 36

Slide 36 text

ֶशɾධՁϓϩηε train & eval process ֶशσʔληοτ ධՁσʔληοτ ࠷ྑύϥϝʔλʔ ߋ৽

Slide 37

Slide 37 text

;͘Ε͕͋Δσʔληοτ

Slide 38

Slide 38 text

Slide 39

Slide 39 text

σʔληοτ؅ཧαʔόʔ ໿ສຕͷը૾ϑΝΠϧʢ໿5#ʣ ໿ສ݅ͷϥϕϧ όοΫΞοϓମ੍ͷ੔උ

Slide 40

Slide 40 text

Slide 41

Slide 41 text

ٕज़తෛ࠴Λͳͥղܾ͠ͳ͍ͷ͔ ͍ͭ·Ͱ΍ΔʁɹͲ͜·Ͱ΍Δʁɹকདྷͷల๬͸ʁ ҆ఆͯ͠Քಇ͍ͯ͠ΔγεςϜΛյͨ͘͠ͳ͍  ʢJGJUXPSLT EPOUpYJUʣ όοΫΞοϓϓϥϯΛ༻ҙͰ͖ͳ͍ʢܦࡁ߹ཧੑʣ

Slide 42

Slide 42 text

͍ͭ·Ͱ΍ΔʁɹͲ͜·Ͱ΍Δʁɹকདྷͷల๬͸ʁ

Slide 43

Slide 43 text

C-LIS CO., LTD. ຊࢿྉ͸ɺ༗ݶձࣾγʔϦεͷஶ࡞෺Ͱ͢ɻຊࢿྉͷશ෦ɺ·ͨ͸Ұ෦ʹ͍ͭͯɺஶ࡞ऀ͔ΒจॻʹΑΔڐ୚Λಘͣʹෳ੡͢Δ͜ͱ͸ې͡ΒΕ͍ͯ·͢ɻ 5IF"OESPJE4UVEJPJDPOJTSFQSPEVDFEPSNPEJpFEGSPNXPSLDSFBUFEBOETIBSFECZ(PPHMFBOEVTFEBDDPSEJOHUPUFSNTEFTDSJCFEJOUIF$SFBUJWF$PNNPOT"UUSJCVUJPO-JDFOTF ֤੡඼໊ɾϒϥϯυ໊ɺձ໊ࣾͳͲ͸ɺҰൠʹ֤ࣾͷ঎ඪ·ͨ͸ొ࿥঎ඪͰ͢ɻຊࢿྉதͰ͸ɺɺɺäΛׂѪ͍ͯ͠·͢ɻ 5IF"OESPJESPCPUJTSFQSPEVDFEPSNPEJpFEGSPNXPSLDSFBUFEBOETIBSFECZ(PPHMFBOEVTFEBDDPSEJOHUPUFSNTEFTDSJCFEJOUIF$SFBUJWF$PNNPOT"UUSJCVUJPO-JDFOTF