$30 off During Our Annual Pro Sale. View Details »

煩雑な運用を
Goを使って楽にする / Go Conference 2019 Spring

Konboi
May 18, 2019

煩雑な運用を
Goを使って楽にする / Go Conference 2019 Spring

GoCon 2019 で発表した 煩雑な運用を
Goを使って楽にする の発表資料になります

Konboi

May 18, 2019
Tweet

More Decks by Konboi

Other Decks in Technology

Transcript

  1. ൥ࡶͳӡ༻Λ

    GoΛ࢖ָͬͯʹ͢Δ
    GoCon2019 Spring / 2019.05.18

    View Slide

  2. About me
    • ໼ਧྒྷհ a.k.a Konboi
    • גࣜձࣾmerpay
    • Coupon Team
    • Backend / Tech Lead

    View Slide

  3. About Talk
    • ӡ༻Ͱൃੜ͢ΔλεΫΛ

    ࣗಈԽ/ޮ཰Խ͢ΔաఔͰͷࢼߦࡨޡͨ͠࿩
    • αʔόʔ/ΠϯϑϥدΓͷ࿩͸͋·Γग़·ͤΜ
    • ஍ຯͰ͢

    View Slide

  4. Agenda
    • GoΛ࢖ͬͯExcelΛѻ͏
    • GoͰGoogleDriveΛѻ͏
    • GoͰେྔͷϑΝΠϧΛੜ੒͢Δ

    View Slide

  5. ͸͡Ίʹ
    • ݱ࣌఺ͰΫʔϙϯσʔλ͸ExcelΛ࢖ͬͯೖߘ
    • ઐ༻ͷೖߘγεςϜ࡞Δͷ͸ίετେ
    • Θ͔Γ΍͍͢UI
    • ηΩϡϦςΟ
    • ద੾ͳݖݶ؅ཧ
    • γεςϜͷӡ༻
    • etc
    • ϛχϚϜͰݕূͯ͠Ձ஋͕͋ΔͳΒ͖ͪΜͱ੔උ

    View Slide

  6. ͸͡Ίʹ
    • ΋ΒͬͨExcel͔ΒखͰσʔλΛ͍͡Δͱ…
    • ώϡʔϚϯΤϥʔ
    • ҙਤ͠ͳ͍σʔλͷࠞೖ
    • ՄೳͳݶΓਓʹΑΔૢ࡞ΛݮΒ͍ͨ͠

    View Slide

  7. ͸͡Ίʹ
    • Execl͸ڞ༗ͷ৔ॴͰ؅ཧ͍ͨ͠
    • ʓʓ͞ΜͷPCʹ͔͠ͳ͍
    • όʔδϣϯ؅ཧ͍ͨ͠
    • 2019/05/18 2.xlsx

    View Slide

  8. ͸͡Ίʹ
    • Google DriveͰ؅ཧ
    • G Suite
    • όʔδϣϯΛ؅ཧ

    View Slide

  9. ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  10. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  11. GoΛ࢖ͬͯExcelΛѻ͏

    View Slide

  12. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  13. GoΛ࢖ͬͯExcelΛѻ͏
    • ద੾ʹσʔλΛநग़͍ͨ͠
    • ద੾ͳϑΥʔϚοτͰѻ͍͍ͨ
    • Project͕GoͳͷͰGoͰExcelΛѻ͍͍ͨ

    View Slide

  14. GoΛ࢖ͬͯExcelΛѻ͏
    • Excel͸ॊೈʹϑΥʔϚοτΛมߋͰ͖Δ
    • ॊೈͳΏ͑ͷ՝୊
    • ηϧ݁߹
    • ؔ਺ʹΑΔೖྗ
    • ॻࣜ
    • etc

    View Slide

  15. ௐࠪ/ݕূ

    View Slide

  16. ௐࠪ/ݕূ
    • ࣮ӡ༻ͷલʹௐࠪ/ݕূΛ࣮ࢪ
    • ͍͔ͭ͘ͷϥΠϒϥϦΛ࢖༻ͯ͠

    ߦͬͨݕূͷҰ෦Λ঺հ͠·͢

    View Slide

  17. ௐࠪ/ݕূ

    View Slide

  18. ௐࠪ/ݕূ
    • ηϧ݁߹

    View Slide

  19. ௐࠪ/ݕূ
    • ηϧ݁߹
    • ࠨ্ͷηϧͷΈσʔλ͕ೖͬͨঢ়ଶʹ
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 06:53,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 06:53,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 06:53,B,

    View Slide

  20. ௐࠪ/ݕূ
    • ؔ਺

    View Slide

  21. ௐࠪ/ݕূ
    • ؔ਺
    • දࣔ͞Ε͍ͯΔσʔλ͕औಘͰ͖Δ
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 06:53,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 06:53,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 06:53,B,

    View Slide

  22. ௐࠪ/ݕূ
    • ؔ਺

    View Slide

  23. ௐࠪ/ݕূ
    • ؔ਺
    • ݱࡏηϧʹೖྗ͞Ε͍ͯΔσʔλΛऔಘ
    $ date "+%Y/%m/%d %H:%M:%S"
    2019/05/18 06:58:14


    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 06:53,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 06:53,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 06:53,B,

    View Slide

  24. ௐࠪ/ݕূ
    • ೖྗܗࣜ
    • Ex)Ϧετ

    View Slide

  25. ௐࠪ/ݕূ
    • ೖྗܗࣜ - Ϧετ
    • ݱࡏηϧʹೖྗ͞Ε͍ͯΔσʔλΛऔಘ
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 06:53,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 06:53,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 06:53,B,

    View Slide

  26. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • Ex) ೔෇

    View Slide

  27. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • ϑΥʔϚοτ͕͓͔͍͠
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 06:53,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 06:53,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 06:53,B,

    View Slide

  28. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • ೔෇͸Excel಺෦ͰγϦΞϧ஋ͱͯ͠อଘ
    • ੔਺஋͕೔෇
    • গ਺஋͕࣌ࠁ
    • ϥΠϒϥϦʹΑͬͯ͸γϦΞϧ஋͕

    ͦͷ··ग़ྗ͞ΕΔ΋ͷ΋

    View Slide

  29. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • ೔෇͸Excel಺෦ͰγϦΞϧ஋ͱͯ͠อଘ

    View Slide

  30. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • ςΩετ

    View Slide

  31. ௐࠪ/ݕূ
    • ॻࣜઃఆ
    • ςΩετ
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,1989/01/08,ఱߖ,ฏ੒:heisei,5/18/19 07:06,A,
    ฏ੒,heisei,2019/04/30,ఱߖ,ฏ੒:heisei,5/18/19 07:06,,
    ྩ࿨,reiwa,2019/05/01,্ߖ,ྩ࿨:reiwa,5/18/19 07:06,B,

    View Slide

  32. ௐࠪ/ݕূ ݁Ռ
    • ΍͔͍ͬͳͷ͸ηϧ݁߹ͱॻࣜઃఆ
    • ͦΕҎ֎͸ҙ֎ͱ໰୊ͳͦ͞͏

    View Slide

  33. ௐࠪ/ݕূ ݁Ռ
    • ͢΂ͯจࣈྻͰ΍Ε͹໰୊͸ͳ͍
    • ໰୊͸ͳ͍…

    View Slide

  34. ௐࠪ/ݕূ ݁Ռ
    • ͢΂ͯจࣈྻͩͱ…
    • શ֯/൒֯ͷѻ͍Λߟྀ
    • ೔ʹͪͳͲΛϑΥʔϚοτ͕͋Δ΋ͷ͸

    ॻࣜઃఆΛઃఆ͠ద੾ͳ஋Λೖྗ/औಘ͍ͨ͠

    View Slide

  35. ௐࠪ/ݕূ ݁Ռ
    • 360EntSecGroup-Skylar/excelize Λ࠾༻
    • SetCellStyle method
    • ॻࣜઃఆΛઃఆͰ͖Δ

    View Slide

  36. • Ex) 14: m/d/yy
    • Ex) 15: d-mmm-yy
    ௐࠪ/ݕূ ݁Ռ
    $ go run main.go -f ../example.xlsx
    name,name_en,date,status,id,created_at,category,
    ฏ੒,heisei,8-Jan-89,ఱߖ,ฏ੒:heisei,5/18/19 07:09,A,
    ฏ੒,heisei,30-Apr-19,ఱߖ,ฏ੒:heisei,5/18/19 07:09,,
    ྩ࿨,reiwa,1-May-19,্ߖ,ྩ࿨:reiwa,5/18/19 07:09,B,
    $ go run main.go -f ../example.xlsx
    ฏ੒,heisei,01-08-89,ఱߖ,ฏ੒:heisei,5/18/19 07:09,A,
    ฏ੒,heisei,04-30-19,ఱߖ,ฏ੒:heisei,5/18/19 07:09,,
    ྩ࿨,reiwa,05-01-19,্ߖ,ྩ࿨:reiwa,5/18/19 07:09,B,

    View Slide

  37. ௐࠪ/ݕূ ݁Ռ
    • ଞʹ΋ExcelΛGo͔Βૢ࡞Մೳ
    • RemoveRow()
    • SaveAs()
    • ExcelʹσʔλΛ௥ه/मਖ਼ͯ͠อଘ͕Մೳ
    • ೖߘεςʔλεͷ؅ཧ͕ExcelͰՄೳʹ

    View Slide

  38. ௐࠪ/ݕূ ݁Ռ
    • จࣈྻ/୯७ͳσʔλͰ͋Ε͹

    ͲͷϥΠϒϥϦͰ΋େମ૝ఆ௨Γʹʹಈ࡞
    • ༻్ʹԠͨ͡ϥΠϒϥϦΛ࢖͏ͷ͕Α͍͔ͱ

    View Slide

  39. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  40. GoΛ࢖ͬͯ

    Google DriveΛѻ͏

    View Slide

  41. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  42. GoͰGoogleDriveΛѻ͏
    • ެࣜͷSDK
    • google.golang.org/api/drive/v3
    • جຊ͸ެࣜυΩϡϝϯτΛݟΕ͹OK
    • https://developers.google.com/drive/api/v3/
    reference/
    • ݸਓతʹቕͬͨͱ͜ΖΛ঺հ


    View Slide

  43. ϑΝΠϧϦετ͸FieldΛࢦఆ
    • ࢦఆ͠ͳ͍ͱid,name, kind, mimeTypeͷΈ
    • ˎࢦఆ͢Δͱ͢΂ͯͷσʔλ͕

    ฦͬͯ͘Δ͔ύϑΥʔϚϯε͕ྼԽ

    srv.Files.List().
    Fields(“files(parents, id, name, owners, kind, mimeType)").
    Do()


    srv.Files.List().Fields(“*”).Do()

    View Slide

  44. Update͢Δͱ͖ͷtips
    • List APIܦ༝ͰFile ObjectΛऔಘ
    • ಛఆͷFieldΛߋ৽ͯ͠Update͢Δͱ
    list, err := service.Files.List().Context(ctx).Do()
    if err != nil {
    return err
    }

    file := list[0]

    file.Name = “dummy”

    _, err := service.Files.Update(file.Id, file).Context(ctx).Do()
    if err != nil {
    return errors.Wrap(err, "error updated")
    }


    > The resource body includes fields which are not directly writable.

    View Slide

  45. Update͢Δͱ͖ͷtips
    • APIͰ͸ߋ৽Ͱ͖ͳ͍Field΋ߋ৽͕͔͔Δ
    • ߋ৽͍ͨ͠FiledΛࢦఆ
    list, err := service.Files.List().Context(ctx).Do()
    if err != nil {
    return err
    }


    file := list[0]

    _, err := service.Files.Update(file.Id, &drive.File{
    Name: “dummy”,
    }).Context(ctx).Do()

    if err != nil {
    return errors.Wrap(err, "error updated")
    }

    View Slide

  46. Update͢Δͱ͖ͷtips
    • ϑΝΠϧͷத਎Λߋ৽
    • Media(io.Reader, Content-Type) Λ࢖༻
    list, err := service.Files.List().Context(ctx).Do()
    if err != nil {
    return err
    }


    file := list[0]

    content := &bytes.Buffer{}
    content.WriteString(“hello”)

    updatedFile, err := service.Files.Update(file.Id, &drive.File{}).
    Media(content, googleapi.ContentType(file.MimeType)).Context(ctx).Do()

    View Slide

  47. Update͢Δͱ͖ͷtips
    • όʔδϣϯ؅ཧ͞ΕΔͷͰ

    ҆৺(?)࣮ͯ͠ߦՄೳ

    View Slide

  48. ࠶ؼతʹϑΝΠϧΛऔಘͰ͖ͳ͍
    • v2ͩͱ Children: list ͱ͍͏API͕͋Δ
    • FileTypeΛνΣοΫͯࣗ͠લͰ

    ࣮૷͢Δඞཁ͕͋Δ
    • .Q(“xxx in parents”)
    • ਌ࢠؔ܎Λࢦఆ

    View Slide

  49. ࠶ؼతʹϑΝΠϧΛऔಘͰ͖ͳ͍
    • ৄ͘͠͸ Google Drive API v3 Ͱݕࡧ
    • sample: Konboi/go-google-drive-example

    list, err := service.Files.List().Q(fmt.Sprintf("'%s' in parents", driveID)).Context(ctx).Do()
    if err != nil {
    return errors.Wrap(err, "list error")
    }


    typeFolder := "application/vnd.google-apps.folder"
    for _, f := range list.Files {
    pp.Printf("file id:%s\nname%s\ntype:%v\n", f.Id, f.Name, f.MimeType)
    if f.MimeType == typeFolder {
    err := ListRecursive(ctx, service, f.Id)
    if err != nil {
    return err
    }
    }
    }

    View Slide

  50. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  51. YAML
    ࣮ݱ͍ͨ͜͠ͱ

    View Slide

  52. GoͰେྔͷϑΝΠϧΛ

    ੜ੒͢Δ

    View Slide

  53. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ

    View Slide

  54. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • ΫʔϙϯʹΑͬͯ͸֤ϢʔβϢχʔΫͳ

    ίʔυΛൃߦ͢Δඞཁ͕͋Δ
    • ༧Ί࣮ࢪళฮ͔ΒίʔυΛൃߦ
    • ݱঢ়ͷγεςϜͷ౎߹্ɺࣄલʹը૾Λ

    ੜ੒͠഑෍͢Δ࢓૊Έ
    • ࣄલʹେྔͷը૾Λੜ੒͢Δඞཁ͕͋Δ

    View Slide

  55. গྔͳΒ໰୊ͳ͍͕

    View Slide

  56. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • ̍ຕ 0.1sec
    • 10ຕ 1sec
    • 1000ຕ 100sec
    • 1ສຕ 1000sec (2.5h)
    • 100ສຕ 100000sec (27h)
    • 1000ສຕ 1000000sec (270h)
    • …

    View Slide

  57. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • ̍ຕ 100kb
    • 10ຕ 1000kb
    • 1000ຕ 100MB
    • 1ສຕ 1GB
    • 100ສຕ 100GB
    • 1000ສຕ 1TB
    • …

    View Slide

  58. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • goroutineͰ΋ݶք͕…
    • ڧ͍ΠϯελϯεͰ࡞Δ͜ͱ΋ߟ͕͑ͨ
    • ౎౓ launch > deploy > shutdown …

    View Slide

  59. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ

    View Slide

  60. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • Google Cloud Functionͱ͸
    • ΠϕϯτۦಈܕͷαʔόʔϨε 

    ίϯϐϡʔςΟϯά ϓϥοτϑΥʔϜ
    • FaaS
    • AWS Lambda

    View Slide

  61. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    ?code=abcd123

    ?code=efgh456

    ?code=abab1212

    View Slide

  62. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    func GenCode(w http.ResponseWriter, r *http.Request) {
    v := r.URL.Query()
    code := v.Get("code")
    img, err := genCode(code, baseImage)
    if err != nil {
    return
    }
    buf := &bytes.Buffer{}
    if err := png.Encode(buf, img); err != nil {
    return
    }
    if err := upload(r.Context(), buf, filename); err != nil {
    return
    }
    fmt.Fprintf(w, "OK")
    w.WriteHeader(http.StatusOK)
    }

    View Slide

  63. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    $ zip -r function.zip function.go some.png vendor
    $ gcloud beta functions deploy \
    --project=${PROJECT_ID} \
    --region=${REGION} \
    --memory=${MEMORY} \
    --runtime=${RUNTIME} \
    --service-account=${SERVICE_ACCOUNT} \
    --source=gs://${UPLOAD_BUCKET_PATH}/${FUNCTION}.zip \
    ${FUNCTION} \
    --entry-point=${FUNCTION} \
    —trigger-http

    View Slide

  64. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    • func (w http.Writer, r *http.Request) Λ

    ΄Μͱͦͷ··ΞοϓϩʔυͰ͖Δ
    • ґଘpackage΋ҰॹʹzipʹؚΊΕ͹࢖༻Մೳ
    • assetϑΝΠϧͳͲ΋ಉࠝՄೳ
    • zipϑΝΠϧ͓Αͼղౚͨ͠ͱ͖ͷ༰ྔ੍ݶ༗

    View Slide

  65. GoͰେྔͷϑΝΠϧΛੜ੒͢Δ
    ?code=abcd123

    ?code=efgh456

    ?code=abab1212

    View Slide

  66. ·ͱΊ
    • ӡ༻Ͱൃੜ͢ΔλεΫΛ

    ࣗಈԽ/ޮ཰Խ͢Δաఔͷࢼߦࡨޡͨ͠࿩
    • GoΛ࢖ͬͯExcelΛѻ͏
    • GoͰGoogleDriveΛѻ͏
    • GoͰେྔͷϑΝΠϧΛੜ੒͢Δ

    View Slide