Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TDD applied Data Cleansing

dproject21
September 08, 2018

TDD applied Data Cleansing

XP祭り2018 LT祭り にて発表

TDDもせずに "AI" とは何事だ?
機械学習ではなく、機械学習で用いるデータのクレンジングにTDDを適用した話をしました。

dproject21

September 08, 2018
Tweet

More Decks by dproject21

Other Decks in Programming

Transcript

  1. 1
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    Inspired by @iwaoRd

    View full-size slide

  2. 2
    ϓϩϑΟʔϧ
    ా໺ޱେथ %BJLJ5BOPHVDIJ !EQSPKFDU

    ϛʔΧϯύχʔגࣜձࣾ
    (PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%

    ྵࡉιϑτϋ΢ε 7#/&5
    4*FS
    8FCαʔϏεاۀ
    ྲྀ࿘ͷຽ
    $+BWB
    ػցֶश *P5τϥΠΞϧࢀը 1ZUIPO

    (PMBOH લॲཧϥΠϒϥϦ։ൃͳͲ
    "LJCBHP
    $PEFGPS+BQBO
    Ϧʔμʔक़
    %FW-07& 1VC
    50$G& #PPUDBNQ
    8"$"5&
    ϛʔΧϯύχʔגࣜձࣾ
    ೖ͙ͬͯ͢ɺϦʔϚϯγϣοΫͰ
    ֎ʹग़Εͳ͔ͬͨ l຤୺ड͚z
    91ࡇΓॳࢀՃ

    View full-size slide

  3. 3
    ϓϩϑΟʔϧ
    ా໺ޱେथ %BJLJ5BOPHVDIJ !EQSPKFDU

    ϛʔΧϯύχʔגࣜձࣾ
    (PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%

    ྵࡉιϑτϋ΢ε 7#/&5
    4*FS
    8FCαʔϏεاۀ
    ྲྀ࿘ͷຽ
    $+BWB
    ػցֶश *P5τϥΠΞϧࢀը 1ZUIPO

    (PMBOH લॲཧϥΠϒϥϦ։ൃͳͲ
    "LJCBHP
    $PEFGPS+BQBO
    Ϧʔμʔक़
    %FW-07& 1VC
    50$G& #PPUDBNQ
    8"$"5&
    ϛʔΧϯύχʔגࣜձࣾ
    ೖ͙ͬͯ͢ɺϦʔϚϯγϣοΫͰ
    ֎ʹग़Εͳ͔ͬͨ l຤୺ड͚z
    91ࡇΓॳࢀՃ

    ͢΂ͯͷόουσʔλΛ
    ੜ·ΕΔલʹফ͠ڈΓ͍ͨ
    ͤΊͯ"*#*ʹ৯ΘͤΔલʹ
    ফ͠ڈΓ͍ͨ
    ࠷ۙͷϙϦγʔ

    View full-size slide

  4. mecompany Inc. 4
    8
    C:
    %
    #
    G+K
    $/&
    /0B
    — „ž’‘ -!
    8
    XVWV WX)Y(
    ,J'EZSYYSXQ
    – …šœ Œ†ƒXeQ
    3 I ¡A>#¢
    P* 9 ¡bci¢
    WRVVV
    W_
    51NŸ?ŸD Ž“ Œ€
    œš Š›žM6ŸF
    ldg
    ean
    mjg
    "L@
    B"
    VYS\Z[]S^[]_
    VYS\\\]SZ_XX
    t||y`UUvrpxvyow~Tvr
    t||y`UU{p}ruTvr
    t||y`UU{p}ruqo|oTvr
    t||y`UU{p}ruZvzTvr
    L@
    ?Ÿ51™ „
    U7DU? =Ÿ2H /
    /ˆž‰‚ž‡ /
    fl /Ÿ™‚ / ŸŸ;
    ŸŸŸ /• ˜” ‹
    ŸŸŸ 4OŸ?.<
    ŸŸŸ kbmdg Ž“ Œ
    ŸŸŸ kbmdgQsxzQhj
    2018 9)8(
    ϓϩϑΟʔϧ

    View full-size slide

  5. mecompany Inc. 5
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    ϓϩϑΟʔϧ

    View full-size slide

  6. mecompany Inc.
    6
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    ػցֶशͷ࿩͸
    ΄΅͠·ͤΜ

    View full-size slide

  7. mecompany Inc.
    7
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    ςετۦಈ։ൃ
    ͖ͪΜͱಡΜͰ
    ͳ͔ͬͨͰ͢
    ͍͢·ͤΜ

    View full-size slide

  8. <3S
    mecompany Inc. 8
    7> QP#?
    y!9/w
    (Az
    <[R'_Uw4KwF6_HX
    Z^\Wkxg* Y^]r`xmoic_\'_%X
    y!9/w
    (Az
    'U
    Cefjt@ ;B.E@
    ;3uxbx
    ;3S MJ O2(
    SCUEL Projectpxlnx
    VL
    $=Ndxqf
    I8G
    &G
    ;wMwOdxqdx
    wwA
    1Dsvoah
    M0
    WEB
    WEB
    WEB
    SCUEL Projectpxlnx
    kxg58
    )+
    :T]
    HP
    2018"9-8,
    ϓϩϑΟʔϧ

    View full-size slide

  9. 2+H
    mecompany Inc. 9
    -
    4 FE5
    t/'r !7u
    t/'r !7u
    J
    9`bem6 18&;6
    1+ns\s
    1+H B? D*!
    SCUEL Projectjsghs
    KA 3C_skb
    > .=
    =
    1rBrD_sk_s
    rr7
    ):lpi[d
    B(
    WEB
    WEB
    WEB
    SCUEL Projectjsghs
    fsc,.
    "#
    0IW
    HP
    20189%8$
    G Z<NYOXV@TUPO
    fsc]oqaq^VQMRWASNL
    ϓϩϑΟʔϧ

    View full-size slide

  10. mecompany Inc. 10
    ެతͳެ։৘ใ͔Β z࢖͑Δzσʔλʹ࢓্ཱͯ͛ͨ
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    ϓϩϑΟʔϧ

    View full-size slide

  11. 5-K
    ࠓ೔ͷ࿩͸ʜ
    mecompany Inc. 11
    0
    7 IH8
    r2)p #:s
    r2)p #:s
    "M
    9
    4-nq_q
    4-K EB G,#
    SCUEL Projectjqghq
    ND 6F`qkb
    A1@
    !@
    4pEpG`qk`q
    pp:
    +=loi^d
    E*
    WEB
    WEB
    WEB
    SCUEL Projectjqghq
    fqc.1
    $%
    3LZ
    HP
    20189'8&
    J"]?Q\R[Y CWXSR
    fqc /YTPUZDVQO
    #*޲͚ͷ
    σʔλ
    ΫϨϯδϯάͷ
    ࿩Ͱ͢

    View full-size slide

  12. 5-K
    ࠓ೔ͷ࿩͸ʜ
    mecompany Inc. 12
    0
    7 IH8
    r2)p #:s
    r2)p #:s
    "M
    9
    4-nq_q
    4-K EB G,#
    SCUEL Projectjqghq
    ND 6F`qkb
    A1@
    !@
    4pEpG`qk`q
    pp:
    +=loi^d
    E*
    WEB
    WEB
    WEB
    SCUEL Projectjqghq
    fqc.1
    $%
    3LZ
    HP
    20189'8&
    J"]?Q\R[Y CWXSR
    fqc /YTPUZDVQO
    ࣗવݴޠॲཧ
    دΓͷ࿩Ͱ͢

    View full-size slide

  13. 13
    • σʔλͦͷ΋ͷͷ඼࣭Λ
    Ҿ্͖͛ΔͨΊʹ
    σʔλΫϨϯδϯάΛ͢Δ
    • σʔλΫϨϯδϯάΛ
    ͢ΔͨΊʹϓϩάϥϜΛॻ͘
    • σʔλΫϨϯδϯάϓϩάϥϜͷ
    ඼࣭Λ୲อ͢ΔͨΊʹ
    ςετίʔυ΋ॻ͘
    z࢖͑ΔzσʔλΛ࢓ཱͯΔͨΊʹ

    View full-size slide

  14. 14

    EC UFDITIPXDBTF
    %ηογϣϯ
    https://www.db-tech-showcase.com/dbts/tokyo
    r
    $PEF GPS +BQBO 4VNNJU ৽ׁ
    lόουΦʔϓϯσʔλڙཆࣉz
    ηογϣϯ
    https://summit2018.code4japan.org/session/210/
    Ͱ׬શ൛Λ΍Γ·͢

    View full-size slide

  15. 201898 15
    σʔλ։ൃͷݱ৔

    View full-size slide


  16. mecompany Inc. 16






    201898
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλ &YDFM $47ͷଞ
    1%' 8FCαΠτ ࢴʢ෺ཧʣ͕͋Δ
    ࢴͱઓ͏ͨΊʹ0$3Λ࢖ͬͨ͋ͱ͸
    ޡೝࣝσʔλͱઓΘͳ͚Ε͹͍͚ͳ͍ɻ
    σʔλ։ൃͷݱ৔

    View full-size slide


  17. mecompany Inc. 17






    201898
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλ
    தʹ͸
    zόουσʔλz ΋ଘࡏ͢Δ
    • ޡࣈ୤ࣈ
    • දهΏΕ
    • ҟମࣈ
    • ༣ศ൪߸ؒҧ͍
    • ి࿩൪߸ ͳ͔ͥ&ϝʔϧΞυϨε
    • ʙࢢʙ۠ ͷ͕۠ൈ͚͍ͯΔ
    σʔλ։ൃͷݱ৔

    View full-size slide

  18. 201898 19
    ॅॴ ۽ຊࢢձ෋ொYYY

    View full-size slide

  19. 201898 20
    ۽ຊࢢձ෋ொ
    ೥ʹ
    ੓ྩࢦఆ౎ࢢ΁Ҡߦ

    View full-size slide

  20. 201898 21
    ۽ຊࢢձ෋ொ
    ۽ຊࢢೆ۠ձ෋ொ
    ˓

    View full-size slide

  21. 201898 22
    5FMYYZZZ[[[[ʲNBJMYQKVH!FYBNQMFDPNʳ

    View full-size slide

  22. 201898 23
    5FMYYZZZ[[[[ʲNBJMYQKVH!FYBNQMFDPNʳ
    ͳΜͰ
    ϝʔϧΞυϨε
    ೖ͍ͬͯΔΜͰ͢ʁ

    View full-size slide

  23. 201898 24
    ͖Ε͍ͳσʔλʹ
    ͯ͋͛͠ͳ͍ͱ
    ෼ੳਫ਼౓͕
    ޲্͠ͳ͍

    View full-size slide

  24. 201898 25
    ͦ͜Ͱ
    5%%Ͱ͢Α

    View full-size slide

  25. https://www.slideshare.net/t_wada/tdd-agile-samurai-base-camp

    View full-size slide

  26. σʔλΫϨϯδϯάϓϩάϥϜ
    ͷ։ൃαΠΫϧ
    ϚτϞͳσʔλΛߟ͑Δ
    ϚτϞͳσʔλΛzҙਤͨ͠ܗࣜzʹม׵ͨ͠
    ݁ՌΛςετέʔεͱͯ͠௥Ճ͢Δ
    ͦͷςετΛ࣮ߦࣦͯ͠ഊͤ͞Δ 3FE

    lҙਤͨ͠ܗࣜzʹม׵͢ΔίʔυΛॻ͘
    Ͱॻ͍ͨςετΛ੒ޭͤ͞Δ (SFFO

    ςετ͕௨Δ··ͰϦϑΝΫλϦϯάΛߦ͏
    3FGBDUPS

    ߥΕͨσʔλΛ௥Ճͯ͠ʙΛ܁Γฦ͢

    View full-size slide

  27. 28
    import (
    "testing"
    "github.com/stretchr/testify/assert”
    )
    func TestCleansingExample (t *testing.T) {
    var testCases := []struct {
    desc string
    in string
    want string
    }{
    {“ϚτϞͳσʔλ”,
    “03-1234-5678”,
    “03-1234-5678”},
    {“ϝΞυೖͬͯΔߥΕͨσʔλ”,
    “TEL:03-1234-9876 email:[email protected]”,
    “03-1234-9876”},
    }
    for _, test := range testCases {
    assert.Equal(t, Cleansing(test.in), test.want, test.desc)
    }
    }
    ͜͜ʹ
    lߥΕͨσʔλzͱ
    l๬·͍͠σʔλzͷ
    ςετέʔεΛՃ͑ͯɺ
    5%%ͷαΠΫϧΛճ͢

    View full-size slide

  28. 29
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    2017
    σʔλ
    ΫϨϯδϯά΋
    ͜ͷαΠΫϧΛ
    ճ͢

    View full-size slide

  29. ͜ͷಓͷΓͰσʔλΫϨϯδϯά
    ϓϩάϥϜΛҭͯͯʜ

    View full-size slide

  30. ෼ੳͰ͖Δɺ͖Ε͍ͳσʔλ΁

    ͜ͷಓͷΓͰߴ͍ਫ਼౓ͷ
    σʔληοτΛ࡞Δ

    View full-size slide

  31. !
    mecompany Inc. 32


    / -0
    / -0
    "
    201898
    લॲཧର৅σʔλ ԯ௒߲໨
    ʢ਺ेສϨίʔυ º਺ສ߲໨ʣ
    %'+.*

    +.*,.)
    ࠃ಺๏ਓ ສϨίʔυ
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλΛ౷߹͠
    l࢖͑Δzσʔλʹ͢ΔͨΊɺલॲཧ͕ඞཁ
    #(
    $&
    +.*,.)
    ଞ͔ࣾΒߪೖͨ͠σʔλϕʔε ਺ԯϨίʔυ
    σʔλ։ൃͷݱ৔

    View full-size slide

  32. 33
    େྔͷσʔλΫϨϯδϯάͰ
    ॲཧ݁ՌΛ֬ೝ͢ΔͨΊʹ
    • খ͍͞σʔληοτͰ
    ϑΟʔυόοΫαΠΫϧΛখ͘͞ճ͢
    • ΑΓେ͖͍σʔληοτΛ૬खʹ
    ໷ؒόονͰ$*Λճ͢
    • l໊دͤz΍ՄࢹԽγεςϜ·ͰσʔλΛ
    σϦόϦͯ͠ϑΟʔυόοΫαΠΫϧΛ
    ճ͢

    View full-size slide

  33. 201898 34
    σʔλΫϨϯδϯάΛ
    ͖ͬͪΓ΍ΔͱԿ͕ඒຯ͍͔͠
    • දهΏΕΛഉআͯ͠NFDBCͰ
    ෼͔ͪॻ͖͠΍͍͢σʔλʹͳΔ
    • ༨෼ͳจࣈྻΛഉআͯ͠ɺଞͷ
    σʔληοτͱܨ͛ΒΕΔ
    • ΧςΰϦ෼͚ϥϕϦϯά͕͖Ε͍ʹ
    Ͱ͖ͯɺσʔλ෼ੳਫ਼౓্͕͕Δ

    View full-size slide

  34. 201898 35
    ػցֶशͰ΋
    ਫ਼౓େࣄ
    Ͱ͢ΑͶ

    View full-size slide

  35. 36
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    σʔλͷ࡞ΓࠐΈɺػցֶशͷΠϯϑϥ͸
    ैདྷͷΤϯδχΞϦϯά͕౔୆

    View full-size slide

  36. 37
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    ػցֶशϞσϧ΋ɺσʔλ΋
    ϑΟʔυόοΫΛճͯ͠ҭͯΔ

    View full-size slide

  37. 38
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    ͜ͷ͋ͨΓͷ࿩͸
    lεϚʔτΤεΠʔzΛ
    ཤम͢Δͱ͍͍ͱࢥ͍·͢
    ʢࢲ͕ཤम͍ͨ͠ʣ

    View full-size slide

  38. mecompany Inc. 39
    ެతͳެ։৘ใ͔Β z࢖͑Δzσʔλʹ࢓্ཱͯ͛ͨ
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    σʔλ։ൃͷݱ৔

    View full-size slide

  39. 40
    • σʔλͦͷ΋ͷͷ඼࣭Λ
    Ҿ্͖͛ΔͨΊʹ
    σʔλΫϨϯδϯάΛ͢Δ
    • σʔλΫϨϯδϯάΛ
    ͢ΔͨΊʹϓϩάϥϜΛॻ͘
    • σʔλΫϨϯδϯάϓϩάϥϜͷ
    ඼࣭Λ୲อ͢ΔͨΊʹ
    ςετίʔυ΋ॻ͘
    z࢖͑ΔzσʔλΛ࢓ཱͯΔͨΊʹ

    View full-size slide

  40. 41
    ۩ମతͳऔΓ૊Έ͸

    EC UFDITIPXDBTF
    %ηογϣϯ
    l೔ຊൃʂ.Z42-6%'ͱ(PݴޠͰ
    ࡞ΔϏοάσʔλલॲཧج൫z
    Ͱ͓࿩͠·͢
    https://www.db-tech-showcase.com/dbts/tokyo

    View full-size slide

  41. 42
    ۩ମతͳऔΓ૊Έ͸
    r
    $PEF GPS +BQBO 4VNNJU ৽ׁ
    lόουΦʔϓϯσʔλڙཆࣉz
    Ͱ͓࿩͠·͢
    https://summit2018.code4japan.org/session/210/
    ։ࣔՄೳͳόουσʔλͷ౤ߘ
    ͓଴͓ͪͯ͠Γ·͢

    View full-size slide

  42. 201898 43
    ݁ہͷͱ͜Ζ

    View full-size slide

  43. 201898 44
    ػցֶश͸
    ΞδϟΠϧ։ൃ
    ͕zલఏz

    View full-size slide

  44. 45
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ

    View full-size slide

  45. 201898 46
    lΞδϟΠϧ։ൃ
    ·ͩ΍ͬͯͳ͍z

    View full-size slide

  46. 201898 47
    ͋ͨͬͯ͠
    ͍·͞ο

    View full-size slide

  47. 48
    91ͱ͸
    ιʔγϟϧνΣϯδ
    Ͱ͋Δ
    UPCFDPOUJOVFˠ EC UFDITIPXDBTF

    View full-size slide