Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google Cloud ML を用いた機械学習基盤の構築と運用/pepabo_ml_infrastructure_starchart

Google Cloud ML を用いた機械学習基盤の構築と運用/pepabo_ml_infrastructure_starchart

GCPUG Fukuoka 5th 〜Machine Learning 祭〜
https://gcpugfukuoka.connpass.com/event/46049/

monochromegane

January 28, 2017
Tweet

More Decks by monochromegane

Other Decks in Technology

Transcript

  1. ࡾ୐༔հ / Pepabo R&D Institute, GMO Pepabo, Inc.
    2017.01.28 GCPUG Fukuoka 5th ʙMachine Learning ࡇʙ
    Google Cloud MLΛ༻͍ͨ
    ػցֶशج൫ͷߏஙͱӡ༻

    View full-size slide

  2. ϓϦϯγύϧΤϯδχΞ
    ࡾ୐༔հ!NPOPDISPNFHBOF
    ϖύϘݚڀॴݚڀһ
    IUUQCMPHNPOPDISPNFHBOFDPN

    View full-size slide

  3. ࠓ೔͓࿩͢͠Δ͜ͱ
    •ϖύϘݚڀॴͱɺͳΊΒ͔ͳγεςϜ
    •ͳΊΒ͔ͳγεςϜΛ࣮ݱ͢Δػցֶशج൫
    •Google Cloud ML ͱ StarChart Λ༻͍ͨػցֶशج
    ൫ͷӡ༻
    https://icons8.com/

    View full-size slide

  4. ϖύϘݚڀॴ

    View full-size slide

  5. ϖύϘݚڀॴ ུশʮϖύݚʯ
    ͸ɺࣄۀΛࠩ
    ผԽͰ͖Δٕज़Λ࡞Γग़ͨ͢ΊʹʮͳΊΒ͔
    ͳγεςϜʯͱ͍͏ίϯηϓτͷԼͰݚڀ։
    ൃʹऔΓ૊Ή૊৫Ͱ͢ɻ
    ϖύϘݚڀॴʹ͍ͭͯ

    http://rand.pepabo.com/

    View full-size slide

  6. ͳΊΒ͔ͳγεςϜ

    View full-size slide

  7. γεςϜͷ֤ཁૉ͕໌ࣔతͳૢ࡞Λܦͣʹಛ
    ௃Λೝࣝ͠ɺͦͷಛ௃΍ؔ܎ੑʹج͖ͮɺͦ
    ͷ࣌ʑͷঢ়گʹԠͨ͡࠷దͳαʔϏεΛఏڙ
    ͢Δ
    ͳΊΒ͔ͳγεςϜ
    http://rand.pepabo.com/

    View full-size slide

  8. ػցֶशͱɺͳΊΒ͔ͳγεςϜ

    View full-size slide

  9. WebαʔϏεͷΞΫηε਺Λ༧ଌ͢Δ
    •ैྔ՝ۚͷԾ૝Ϧιʔεӡ༻ʹ͓͍ͯ࠷దͳϦιʔεधཁͷ༧
    ଌ͸ίετΧοτʹͭͳ͕Δ
    •WebαʔϏεͷϦιʔεधཁ͸ϦΫΤετॲཧ݅਺ɺͭ·ΓΞ
    Ϋηε਺ͱ૬͕ؔ͋Δ͸ͣ
    •Ϧιʔεͷ૿ݮʹ΋͋Δఔ౓ͷ͕͔͔࣌ؒΔͨΊɺϦΞϧλΠ
    ϜͰ͸ͳ͘ҰఆִؒͰͷΞΫηε਺༧ଌͰे෼ͱߟ͑Δ

    View full-size slide

  10. WebαʔϏεͷΞΫηε਺Λ༧ଌ͢Δ
    ΞΫηε਺Λ༧ଌͰ͖ΔΑ͏ʹͳΕ
    ͹ɺϐʔΫλΠϜʹ͋Θͤͨ୆਺ݟੵ
    ΋Γ͔Β࣌ؒ͝ͱͷ࠷దʢͱࢥΘΕ
    Δʣ୆਺ݟੵ΋Γ͕ՄೳʹͳΔ

    View full-size slide

  11. LSTM
    Long Short Term Memory

    View full-size slide

  12. WebαʔϏεͷΞΫηε਺Λ༧ଌ͢Δ
    -45.Λ༻͍ͨΞΫηε਺༧ଌ
    w ೖྗ͸ظ ೔෼
    ͷΞΫηε਺ͱΧϨϯμʔ৘ใ
    w ग़ྗ͸༧ଌͨ͠ظઌͷΞΫηε਺
    w ظઌͷ༧ଌʹ͸લճͷ༧ଌΛೖྗʹؚΊͳ͕Β
    ظઌ·Ͱͷ༧ଌ
    w ࠨਤ͸िؒ෼Λ܁Γฦ͠༧ଌ
    IUUQSBOEQFQBCPDPNBSUJDMFNPOPDISPNFHBOF

    View full-size slide

  13. ΞΧσϛοΫͳਫ४ʹ͓͚Δ৽نੑɾ༗ޮੑɾ
    ৴པੑΛ௥ٻ͢ΔݚڀΛߦ͏ͱͱ΋ʹɺݚڀ
    ։ൃٕͨ͠ज़Λ࣮ࡍͷγεςϜͱ࣮ͯ͠૷ɾ
    ఏڙ͢Δ͜ͱΛ௨ͯ͠ɺࣄۀͷ੒௕ʹߩݙ͠
    ·͢ɻ
    ϖύϘݚڀॴʹ͍ͭͯ

    http://rand.pepabo.com/

    View full-size slide

  14. Service meets ML.
    ࢖ͬͯφϯϘ

    View full-size slide

  15. ػցֶशج൫ʹ
    ٻΊΒΕΔ΋ͷ

    View full-size slide

  16. ྫ͑͹ɺΞΫηε਺༧ଌΛαʔϏεͰ࢖͏
    Users Service
    log
    rack-bigfoot
    Bigfoot
    Activity
    ML Platform
    training
    dataset
    train
    input data
    prediction
    tune
    Access count prediction and schedule scaling

    View full-size slide

  17. ྫ͑͹ɺΞΫηε਺༧ଌΛαʔϏεͰ࢖͏
    Users Service
    log
    rack-bigfoot
    Bigfoot
    Activity
    ML Platform
    training
    dataset
    train
    input data
    prediction
    tune
    Access count prediction and schedule scaling
    ͍͔ͳΔಛ௃Λ͔࣋ͭΛ
    ਫ਼៛ʹೝࣝ͢Δ
    ਓؒʹ໌ࣔతͳૢ࡞Λ
    ՝͞ͳ͍
    ͦͷ࣌ʑͷঢ়گʹԠͯ͡
    ࠷దͳαʔϏεΛఏڙ͢Δ

    View full-size slide

  18. OK, ͳΊΒ͔

    View full-size slide

  19. ػցֶशج൫ͷཁ݅Λߟ͑Δ
    ϩά΍%#ͳͲͷαʔϏεࢿ࢈ͱ࿈ܞͰ͖Δ
    ൺֱత༰қʹϞσϧͷߏஙͱࢼߦ͕ߦ͑Δ
    ֶश݁ՌΛར༻͢ΔͨΊͷखஈͱͯ͠"1*Λఏڙ͢Δ
    ֶश݁ՌͷϩʔΧϧར༻͕Ͱ͖Δͱͳ͓Α͍
    ্هͷ࢓૊Έ͕εέʔϥϒϧͰ͋Δ͜ͱ

    View full-size slide

  20. Google Cloud ML Ͱߟ͑Δ
    ೖग़ྗ͕$MPVE4UPSBHFܦ༝
    ܇࿅ϓϩάϥϜͱͯ͠5FOTPS'MPXΛ࠾༻
    ΦϯϥΠϯ༧ଌαʔϏεʹΑΓϞσϧͷ"1*Խ
    ֶश݁Ռ͸$MPVE4UPSBHFʹอଘɺϩʔΧϧͰͷར༻΋
    ෼ࢄܕͷτϨʔχϯάΠϯϑϥͱෛՙ෼ࢄαʔϏεͱͷ࿈ܞ
    ※ ݕ౼ʹؔ͢Δৄࡉ:
    http://rand.pepabo.com/article/2017/01/18/pepabo-ml-platform-and-workflow/

    View full-size slide

  21. ػցֶशج൫Λ
    ӡ༻͢Δ

    View full-size slide

  22. ֶश݁Ռ͸Ӭଓతͳ΋ͷͰ͸ͳ͍
    expose
    Create
    1. ϞσϧΛߏஙͯ͠ެ։͢Δ
    2. αʔϏεࢿ࢈ͷมԽʹ߹ΘͤͯϞσϧΛվળ͢Δ
    tune

    View full-size slide

  23. ͦͷϞσϧɺαʔϏεʹఏڙͯ͠େৎ෉ʁ
    expose
    create
    ֶशͨ͠಺༰͕֬ೝͳ͘ద༻͞Εͯ͠·͏͜ͱͰҙਤ͠ͳ͍݁
    Ռ͕ར༻͞ΕΔ͜ͱΛ๷͍͗ͨ
    tune
    Ϟσϧֶश͠ͳ͓ͨ͠ͷͰɺ
    APIͷ݁Ռ͕มΘͬͯ·͢
    ਫ਼౓Լ͕ͬͯΔؾ͕͢Δʁ
    ग़ྗͷ࣍ݩ਺มΘͬͯͳ͍ʁ

    View full-size slide

  24. ͦͷϞσϧɺαʔϏεʹఏڙͯ͠େৎ෉ʁ
    expose
    create
    ֶशͨ͠಺༰͕֬ೝͳ͘ద༻͞Εͯ͠·͏͜ͱͰҙਤ͠ͳ͍݁
    Ռ͕ར༻͞ΕΔ͜ͱΛ๷͍͗ͨ -> Ϟσϧͷόʔδϣϯ؅ཧ
    tune
    Ϟσϧֶशͯ͠৽όʔδϣϯ
    ͭͬͯ͘·͢ɻ੾ସ͍͍Ͱ͔͢ʁ
    ͍͖ͳΓมΘΒͳͯ͘ศར
    ͚ͩͲɺมߋ಺༰͕Θ͔ΒΜͳʁ
    default

    View full-size slide

  25. ͦͷϞσϧɺαʔϏεʹఏڙͯ͠େৎ෉ʁ
    expose
    create
    ֶशͨ͠಺༰͕֬ೝͳ͘ద༻͞Εͯ͠·͏͜ͱͰҙਤ͠ͳ͍݁
    Ռ͕ར༻͞ΕΔ͜ͱΛ๷͍͗ͨ -> Ϟσϧͷίʔυ؅ཧ
    tune
    Ϟσϧͷ৽όʔδϣϯɺมߋ಺༰ͷ
    ϨϏϡʔ͓ئ͍͠·͢ʂ
    LGTM!!!
    όʔδϣϯ੾Γସ͑·͢ʂʂ
    default
    management

    View full-size slide

  26. StarChart
    https://github.com/monochromegane/starchart

    View full-size slide

  27. StarChart is a tool to manage Google Cloud
    Machine Learning training programs and
    model versions.
    StarChart

    View full-size slide

  28. StarChart
    Train job,
    model default
    version
    Train programs and
    model versions
    on GitHub
    train,
    expose,
    apply
    StarChart
    • όʔδϣϯ؅ཧͷ੾ସʹ͓͚Δ൑அج४ͱͳΔ܇࿅ϓϩάϥϜɺύϥϝλɺδϣϒ
    ৘ใ·ͰؚΊͯίʔυͰ؅ཧ
    • ֶश࣌ͷδϣϒID΍Cloud Storageͷύεɺόʔδϣϯʹඥͮ͘ύϥϝλ৘ใͷऔ
    ಘʹ·ͭΘΔCloud MLͷࡉ͔ͳ࢖͍উख΋վળ

    View full-size slide

  29. DCGAN on Cloud ML
    using StarChart

    View full-size slide

  30. DCGAN
    TensorFlowʹΑΔDCGANͰΞΠ
    υϧͷإը૾ੜ੒
    http://memo.sugyan.com/entry/20160516/1463359395

    View full-size slide

  31. ܇࿅ϓϩάϥϜΛGit؅ཧ͢Δ
    .
    !"" dcgan
    #"" setup.py
    !"" trainer
    #"" __init__.py
    #"" dcgan.py
    !"" task.py
    Ϟσϧ໊ͷ഑Լʹύοέʔδߏ੒ʹͳ
    ΔΑ͏ʹ܇࿅ϓϩάϥϜΛ഑ஔ
    ґଘύοέʔδ͕͋Δ৔߹͸ɺ
    setup.pyΛ४උ
    ࠓճ͸face-generatorͷdcgan.pyͱ
    main.pyΛར༻
    https://github.com/sugyan/face-generator

    View full-size slide

  32. ܇࿅ϓϩάϥϜΛδϣϒͱͯ͠ొ࿥
    $ starchart train \
    -m dcgan \ # MODEL_NAME
    -M trainer.task \ # MODULE_NAME
    -- \
    --train_dir=TRAIN_PATH/model \ # YOUR_TRAIN_PARAMS
    --images_dir=TRAIN_PATH/images \
    --data_dir=gs://$BUCKET_NAME/data/dcgan
    • ύοέʔδϯάɺCloud Storage΁ͷΞοϓϩʔυɺδϣϒొ࿥Λ࣮ߦ
    • `TRAIN_PATH`͸Cloud Storage্ʹδϣϒ͝ͱʹ࡞੒͞ΕΔσΟϨΫτϦ໊ʹղऍ
    • ϓϩδΣΫτIDɺϦʔδϣϯɺΫϨσϯγϟϧ͸direnvܦ༝ͷ؀ڥม਺ࢦఆ͕ศར

    View full-size slide

  33. δϣϒͷ࣮ߦΛ଴ͭ
    $ starchart state -m dcgan
    jobId: dcgan_20170125191521 (FAILED)
    • δϣϒIDͱεςʔλεΛ֬ೝ
    • ϩάදࣔػೳ͸ະ࣮૷

    View full-size slide

  34. ܇࿅ϓϩάϥϜΛ$MPVE.-ʹରԠͤ͞Δ
    import os
    os.listdir()
    os.mkdir(path)
    os.path.exists()
    with open(filename, 'wb') as f:
    • δϣϒ࣮ߦ࣌ͷFileIO͸Cloud StorageΛର৅ͱ͢Δ
    • tensorflow.python.lib.io.file_ioύοέʔδΛ࢖͏͜ͱͰϩʔΧϧύεࢦఆɺCloud
    Storageࢦఆ(gs://)Λಁաతʹѻ͑Δ
    • ύεࢦఆ͸ίϚϯυϥΠϯҾ਺Ͱ౉ͤΔΑ͏࣮૷͓ͯ͘͠ͱศར
    from tensorflow.python.lib.io import file_io
    file_io.list_directory()
    file_io.create_dir(path)
    file_io.file_exists()
    with file_io.FileIO(filename, 'w') as f:

    View full-size slide

  35. ܇࿅ϓϩάϥϜΛδϣϒͱͯ͠ొ࿥
    $ starchart train \
    -m dcgan \ # MODEL_NAME
    -M trainer.task \ # MODULE_NAME
    -- \
    --train_dir=TRAIN_PATH/model \ # YOUR_TRAIN_PARAMS
    --images_dir=TRAIN_PATH/images \
    --data_dir=gs://$BUCKET_NAME/data/dcgan

    View full-size slide

  36. δϣϒͷ࣮ߦΛ଴ͭ
    $ starchart state -m dcgan
    jobId: dcgan_20170125194440 (SUCCESSED)
    jobId: dcgan_20170125191521 (FAILED)
    • δϣϒIDͱεςʔλεΛ֬ೝ

    View full-size slide

  37. ϞσϧΛެ։͢Δ
    $ starchart expose -m dcgan
    • ੒ޭͨ͠δϣϒΛݩʹϞσϧΛొ࿥
    • Ϟσϧొ࿥࣌ʹόʔδϣϯ΋ొ࿥ͯ͠༧ଌαʔϏεAPIͱͯ͠ެ։
    • όʔδϣϯ໊͸ v + δϣϒ໊
    model
    v20170125194440
    (default)
    Cloud ML & Storage

    View full-size slide

  38. Not working…

    View full-size slide

  39. ༧ଌαʔϏε"1*ͷ࢓૊Έʢٖࣅίʔυʣ
    request_params = {'instances': [{'sample_inputs': np.zeros((1, 40)).tolist()}]}
    def feed_from_request(request, tensor_keys):
    feed = {}
    request_keys = request['instances'][0].keys()
    for key in request_keys:
    feed[tensor_keys[key]] = [instance[key] for instance in request['instances']]
    return feed
    with tf.Session() as sess:
    new_saver = tf.train.import_meta_graph(‘TRAIN_PATH/model/export.meta’)
    new_saver.restore(sess, ‘TRAIN_PATH/model/export’)
    tensor_keys = json.loads(tf.get_collection('inputs')[0])
    feed = feed_from_request(request_params, tensor_keys)
    op = json.loads(tf.get_collection('outputs')[0])
    result = sess.run(op, feed_dict=feed)
    print(result)
    ֶश݁Ռͷ.FUB(SBQIΛ෮ݩ
    ίϨΫγϣϯ JOQVUT
    ͷςϯι
    ϧͱ"1*ϦΫΤετύϥϝλΛ
    ඥ෇͚
    ίϨΫγϣϯ PVUQVUT
    ͷςϯ
    ιϧΛΦϖϨʔγϣϯͱͯ͠ඥ
    ෇͚ͨGFFEΛҾ਺ʹ࣮ߦ

    View full-size slide

  40. ܇࿅ϓϩάϥϜΛ"1*ʹରԠͤ͞Δ

    saver.save(sess, os.path.join(FLAGS.train_dir, 'export'))
    • APIͰར༻͢ΔͨΊɺ࠷ऴͷֶश݁ՌΛΤΫεϙʔτ͢Δ
    • ΤΫεϙʔτ໊͸exportͰͳ͚Ε͹ͳΒͳ͍
    • StarChartͷ৔߹ɺ`TRAIN_PATH/model/export` ͱͯ͠ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍

    View full-size slide

  41. • ίϨΫγϣϯʹೖྗ༻ςϯιϧΛ௥Ճɻग़ྗςϯιϧͷ࣮ߦ࣌ʹfeed_dictͱͯ͠౉
    ͢ύϥϝλΛࢦఆ
    • ίϨΫγϣϯʹग़ྗ༻ςϯιϧΛ௥Ճɻ༧ଌαʔϏεAPI࣮ߦ࣌ͷΦϖϨʔγϣϯ
    Λࢦఆ
    • ग़ྗ༻ςϯιϧʹ `tf.image.encode_jpeg`Λ࢖͏ͱInternal Server Error ͩͬͨͷͰtf.reshape(tf.squeeze(image, [0]), [1, -1]) ͱͨ͠
    # Input
    sample_inputs = tf.placeholder(tf.float32, shape=(None, 1, dcgan.z_dim))
    tf.add_to_collection('inputs', json.dumps({'sample_inputs': sample_inputs.name}))
    # Output
    sample_outputs = dcgan.sample_image_vectors(1, 1, inputs=sample_inputs[0])
    tf.add_to_collection('outputs', json.dumps({'sample_outputs': sample_outputs.name}))
    ܇࿅ϓϩάϥϜΛ"1*ʹରԠͤ͞Δ

    View full-size slide

  42. ܇࿅ϓϩάϥϜΛδϣϒͱͯ͠ొ࿥
    $ starchart train \
    -m dcgan \ # MODEL_NAME
    -M trainer.task \ # MODULE_NAME
    -- \
    --train_dir=TRAIN_PATH/model \ # YOUR_TRAIN_PARAMS
    --images_dir=TRAIN_PATH/images \
    --data_dir=gs://$BUCKET_NAME/data/dcgan

    View full-size slide

  43. δϣϒͷ࣮ߦΛ଴ͭ
    $ starchart state -m dcgan
    jobId: dcgan_20170125201233 (SUCCESSED)
    jobId: dcgan_20170125194440 (SUCCESSED)
    jobId: dcgan_20170125191521 (FAILED)
    • δϣϒIDͱεςʔλεΛ֬ೝ

    View full-size slide

  44. ϞσϧΛެ։͢Δ
    $ starchart expose -m dcgan
    model
    v20170125194440
    (default)
    Cloud ML & Storage v20170125201233

    View full-size slide

  45. ϞσϧϑΝΠϧΛGit؅ཧ͢Δ
    .
    #"" dcgan
    $ #"" setup.py
    $ !"" trainer
    $ #"" __init__.py
    $ #"" dcgan.py
    $ !"" task.py
    !"" dcgan.json
    exposeͨ݁͠Ռ͕`Ϟσϧ໊.json`ʹอ
    ଘ͞ΕΔɻσϑΥϧτόʔδϣϯͷ੾
    ସʹ࢖͏ͷͰ͜Ε΋Git؅ཧͱ͢Δ

    View full-size slide

  46. ϞσϧϑΝΠϧΛGit؅ཧ͢Δ
    {
    "model": "MODEL_NAME",
    "versions": [
    {
    "version": {
    "name": "projects/PROJECT_ID/models/MODEL_NAME/versions/v20170111170842",
    "deploymentUri": "gs://PROJECT_ID-ml/MODEL_NAME/20170111170842/model",
    "createTime": "2017-01-11T09:12:54Z",
    "job": {
    "jobId": "MODEL_NAME_20170111170842",
    "trainingInput": {
    "packageUris": [
    "gs://PROJECT_ID-ml/MODEL_NAME/20170111170842/packages/trainer-0.0.0.tar.gz"
    ],
    "pythonModule": "trainer.task",
    "args": [
    "--model_dir=gs://PROJECT_ID-ml/MODEL_NAME/20170111170842/model",
    "--train_dir=gs://PROJECT_ID-ml/MODEL_NAME/20170111170842/train",
    ],
    "region": "us-central1"
    },
    "createTime": "2017-01-11T08:08:49Z",
    "startTime": "2017-01-11T08:13:55Z",
    "endTime": "2017-01-11T08:40:55Z",
    "state": "SUCCEEDED",
    "trainingOutput": {
    "consumedMLUnits": 0.45
    }
    },
    "isDefault": true
    }
    }
    ]
    }
    "1*όʔδϣϯʹඥͮ͘δϣϒ
    ΍࣮ߦ࣌ύϥϝλɺσϑΥϧτ
    όʔδϣϯ͔Ͳ͏͔֬ೝͰ͖Δ

    View full-size slide

  47. ༧ଌαʔϏε"1*Λ࢖͏
    project = 'project-123456'
    model = 'dcgan'
    version = 'v20170125194440'
    credentials = GoogleCredentials.get_application_default()
    ml = discovery.build('ml', 'v1beta1', credentials=credentials)
    body = {'instances': [{'sample_inputs': np.zeros((1, 40)).tolist()}]}
    request = ml.projects().predict(name='projects/{}/models/{}/versions/{}'.format(project, model, version), body=body)
    try:
    response = request.execute()
    output = response['predictions'][0]['sample_outputs']
    with tf.Session() as sess:
    image = sess.run(tf.image.encode_jpeg(tf.reshape(tf.constant(output, dtype=tf.uint8), [96, 96, 3])))
    with file_io.FileIO('out.jpg', 'w') as f:
    f.write(image)
    except errors.HttpError as err:
    print(err._get_reason())

    View full-size slide

  48. ϞσϧͷσϑΥϧτόʔδϣϯΛมߋ͢Δ
    $ starchart apply -m dcgan
    model
    v20170125194440
    Cloud ML & Storage
    v20170125201233
    (default)
    • ϞσϧϑΝΠϧΛฤू͠ɺσϑΥϧτͱ͍ͨ͠όʔδϣϯͷ `isDefault` Λ true ʹɻ
    • ͜ͷ࣌఺ͷ܇࿅ϓϩάϥϜͱϞσϧϑΝΠϧΛPullRequestͱ͢Δ
    • ϨϏϡʔͰσϑΥϧτόʔδϣϯͷج४Λຬ͍ͨͯͨ͠ΒϚʔδͯ͠apply

    View full-size slide

  49. 4UBS$IBSUʹΑΔӡ༻ͷ͓͞Β͍
    •Ϟσϧ͝ͱͷ܇࿅ϓϩάϥϜΛGit؅ཧ
    •train -> expose -> ϨϏϡʔ -> apply Λ܁Γฦ͢
    Easy & Useful

    View full-size slide

  50. ·ͱΊ
    •ػցֶशΛαʔϏεར༻͢ΔͨΊʹػցֶशج൫Λݕ
    ౼ͨ͠
    •Google Cloud MLʹΑΔߏஙɺӡ༻ΛߦͬͯΈͨ
    •ෆศͳ఺͸StarChartͰվળͨ͠
    •ػցֶशͰαʔϏεվળ͠Α͏ʂʂ

    View full-size slide

  51. ϖύϘΧϨοδظੜืूத
    ʙ෱ԬͰ׆༂͍ͨ͠ʂ8FCΞϓϦέʔγϣϯΤϯδχΞʙ
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

    View full-size slide