Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TDD applied Data Cleansing

dproject21
September 08, 2018

TDD applied Data Cleansing

XP祭り2018 LT祭り にて発表

TDDもせずに "AI" とは何事だ?
機械学習ではなく、機械学習で用いるデータのクレンジングにTDDを適用した話をしました。

dproject21

September 08, 2018
Tweet

More Decks by dproject21

Other Decks in Programming

Transcript

  1. 1
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    Inspired by @iwaoRd

    View Slide

  2. 2
    ϓϩϑΟʔϧ
    ా໺ޱେथ %BJLJ5BOPHVDIJ !EQSPKFDU

    ϛʔΧϯύχʔגࣜձࣾ
    (PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%

    ྵࡉιϑτϋ΢ε 7#/&5
    4*FS
    8FCαʔϏεاۀ
    ྲྀ࿘ͷຽ
    $+BWB
    ػցֶश *P5τϥΠΞϧࢀը 1ZUIPO

    (PMBOH લॲཧϥΠϒϥϦ։ൃͳͲ
    "LJCBHP
    $PEFGPS+BQBO
    Ϧʔμʔक़
    %FW-07& 1VC
    50$G& #PPUDBNQ
    8"$"5&
    ϛʔΧϯύχʔגࣜձࣾ
    ೖ͙ͬͯ͢ɺϦʔϚϯγϣοΫͰ
    ֎ʹग़Εͳ͔ͬͨ l຤୺ड͚z
    91ࡇΓॳࢀՃ

    View Slide

  3. 3
    ϓϩϑΟʔϧ
    ా໺ޱେथ %BJLJ5BOPHVDIJ !EQSPKFDU

    ϛʔΧϯύχʔגࣜձࣾ
    (PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%

    ྵࡉιϑτϋ΢ε 7#/&5
    4*FS
    8FCαʔϏεاۀ
    ྲྀ࿘ͷຽ
    $+BWB
    ػցֶश *P5τϥΠΞϧࢀը 1ZUIPO

    (PMBOH લॲཧϥΠϒϥϦ։ൃͳͲ
    "LJCBHP
    $PEFGPS+BQBO
    Ϧʔμʔक़
    %FW-07& 1VC
    50$G& #PPUDBNQ
    8"$"5&
    ϛʔΧϯύχʔגࣜձࣾ
    ೖ͙ͬͯ͢ɺϦʔϚϯγϣοΫͰ
    ֎ʹग़Εͳ͔ͬͨ l຤୺ड͚z
    91ࡇΓॳࢀՃ

    ͢΂ͯͷόουσʔλΛ
    ੜ·ΕΔલʹফ͠ڈΓ͍ͨ
    ͤΊͯ"*#*ʹ৯ΘͤΔલʹ
    ফ͠ڈΓ͍ͨ
    ࠷ۙͷϙϦγʔ

    View Slide

  4. mecompany Inc. 4
    8
    C:
    %
    #
    G+K
    $/&
    /0B
    — „ž’‘ -!
    8
    XVWV WX)Y(
    ,J'EZSYYSXQ
    – …šœ Œ†ƒXeQ
    3 I ¡A>#¢
    P* 9 ¡bci¢
    WRVVV
    W_
    51NŸ?ŸD Ž“ Œ€
    œš Š›žM6ŸF
    ldg
    ean
    mjg
    "[email protected]
    B"
    VYS\Z[]S^[]_
    VYS\\\]SZ_XX
    t||y`UUvrpxvyow~Tvr
    t||y`UU{p}ruTvr
    t||y`UU{p}ruqo|oTvr
    t||y`UU{p}ruZvzTvr
    [email protected]
    ?Ÿ51™ „
    U7DU? =Ÿ2H /
    /ˆž‰‚ž‡ /
    fl /Ÿ™‚ / ŸŸ;
    ŸŸŸ /• ˜” ‹
    ŸŸŸ 4OŸ?.<
    ŸŸŸ kbmdg Ž“ Œ
    ŸŸŸ kbmdgQsxzQhj
    2018 9)8(
    ϓϩϑΟʔϧ

    View Slide

  5. mecompany Inc. 5
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    ϓϩϑΟʔϧ

    View Slide

  6. mecompany Inc.
    6
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    ػցֶशͷ࿩͸
    ΄΅͠·ͤΜ

    View Slide

  7. mecompany Inc.
    7
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ
    ςετۦಈ։ൃ
    ͖ͪΜͱಡΜͰ
    ͳ͔ͬͨͰ͢
    ͍͢·ͤΜ

    View Slide

  8. <3S
    mecompany Inc. 8
    7> QP#?
    y!9/w
    (Az
    <[R'_Uw4KwF6_HX
    Z^\Wkxg* Y^]r`xmoic_\'_%X
    y!9/w
    (Az
    'U
    [email protected] ;[email protected]
    ;3uxbx
    ;3S MJ O2(
    SCUEL Projectpxlnx
    VL
    $=Ndxqf
    I8G
    &G
    ;wMwOdxqdx
    wwA
    1Dsvoah
    M0
    WEB
    WEB
    WEB
    SCUEL Projectpxlnx
    kxg58
    )+
    :T]
    HP
    2018"9-8,
    ϓϩϑΟʔϧ

    View Slide

  9. 2+H
    mecompany Inc. 9
    -
    4 FE5
    t/'r !7u
    t/'r !7u
    J
    9`bem6 18&;6
    1+ns\s
    1+H B? D*!
    SCUEL Projectjsghs
    KA 3C_skb
    > .=
    =
    1rBrD_sk_s
    rr7
    ):lpi[d
    B(
    WEB
    WEB
    WEB
    SCUEL Projectjsghs
    fsc,.
    "#
    0IW
    HP
    20189%8$
    G Z<[email protected]
    fsc]oqaq^VQMRWASNL
    ϓϩϑΟʔϧ

    View Slide

  10. mecompany Inc. 10
    ެతͳެ։৘ใ͔Β z࢖͑Δzσʔλʹ࢓্ཱͯ͛ͨ
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    ϓϩϑΟʔϧ

    View Slide

  11. 5-K
    ࠓ೔ͷ࿩͸ʜ
    mecompany Inc. 11
    0
    7 IH8
    r2)p #:s
    r2)p #:s
    "M
    9
    4-nq_q
    4-K EB G,#
    SCUEL Projectjqghq
    ND 6F`qkb
    [email protected]
    [email protected]
    4pEpG`qk`q
    pp:
    +=loi^d
    E*
    WEB
    WEB
    WEB
    SCUEL Projectjqghq
    fqc.1
    $%
    3LZ
    HP
    20189'8&
    J"]?Q\R[Y CWXSR
    fqc /YTPUZDVQO
    #*޲͚ͷ
    σʔλ
    ΫϨϯδϯάͷ
    ࿩Ͱ͢

    View Slide

  12. 5-K
    ࠓ೔ͷ࿩͸ʜ
    mecompany Inc. 12
    0
    7 IH8
    r2)p #:s
    r2)p #:s
    "M
    9
    4-nq_q
    4-K EB G,#
    SCUEL Projectjqghq
    ND 6F`qkb
    [email protected]
    [email protected]
    4pEpG`qk`q
    pp:
    +=loi^d
    E*
    WEB
    WEB
    WEB
    SCUEL Projectjqghq
    fqc.1
    $%
    3LZ
    HP
    20189'8&
    J"]?Q\R[Y CWXSR
    fqc /YTPUZDVQO
    ࣗવݴޠॲཧ
    دΓͷ࿩Ͱ͢

    View Slide

  13. 13
    • σʔλͦͷ΋ͷͷ඼࣭Λ
    Ҿ্͖͛ΔͨΊʹ
    σʔλΫϨϯδϯάΛ͢Δ
    • σʔλΫϨϯδϯάΛ
    ͢ΔͨΊʹϓϩάϥϜΛॻ͘
    • σʔλΫϨϯδϯάϓϩάϥϜͷ
    ඼࣭Λ୲อ͢ΔͨΊʹ
    ςετίʔυ΋ॻ͘
    z࢖͑ΔzσʔλΛ࢓ཱͯΔͨΊʹ

    View Slide

  14. 14

    EC UFDITIPXDBTF
    %ηογϣϯ
    https://www.db-tech-showcase.com/dbts/tokyo
    r
    $PEF GPS +BQBO 4VNNJU ৽ׁ
    lόουΦʔϓϯσʔλڙཆࣉz
    ηογϣϯ
    https://summit2018.code4japan.org/session/210/
    Ͱ׬શ൛Λ΍Γ·͢

    View Slide

  15. 201898 15
    σʔλ։ൃͷݱ৔

    View Slide


  16. mecompany Inc. 16






    201898
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλ &YDFM $47ͷଞ
    1%' 8FCαΠτ ࢴʢ෺ཧʣ͕͋Δ
    ࢴͱઓ͏ͨΊʹ0$3Λ࢖ͬͨ͋ͱ͸
    ޡೝࣝσʔλͱઓΘͳ͚Ε͹͍͚ͳ͍ɻ
    σʔλ։ൃͷݱ৔

    View Slide


  17. mecompany Inc. 17






    201898
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλ
    தʹ͸
    zόουσʔλz ΋ଘࡏ͢Δ
    • ޡࣈ୤ࣈ
    • දهΏΕ
    • ҟମࣈ
    • ༣ศ൪߸ؒҧ͍
    • ి࿩൪߸ ͳ͔ͥ&ϝʔϧΞυϨε
    • ʙࢢʙ۠ ͷ͕۠ൈ͚͍ͯΔ
    σʔλ։ൃͷݱ৔

    View Slide

  18. 18
    ྫ͑͹ʜ

    View Slide

  19. 201898 19
    ॅॴ ۽ຊࢢձ෋ொYYY

    View Slide

  20. 201898 20
    ۽ຊࢢձ෋ொ
    ೥ʹ
    ੓ྩࢦఆ౎ࢢ΁Ҡߦ

    View Slide

  21. 201898 21
    ۽ຊࢢձ෋ொ
    ۽ຊࢢೆ۠ձ෋ொ
    ˓

    View Slide

  22. 201898 22
    5FMYYZZZ[[[[ʲNBJMYQKVH!FYBNQMFDPNʳ

    View Slide

  23. 201898 23
    5FMYYZZZ[[[[ʲNBJMYQKVH!FYBNQMFDPNʳ
    ͳΜͰ
    ϝʔϧΞυϨε
    ೖ͍ͬͯΔΜͰ͢ʁ

    View Slide

  24. 201898 24
    ͖Ε͍ͳσʔλʹ
    ͯ͋͛͠ͳ͍ͱ
    ෼ੳਫ਼౓͕
    ޲্͠ͳ͍

    View Slide

  25. 201898 25
    ͦ͜Ͱ
    5%%Ͱ͢Α

    View Slide

  26. https://www.slideshare.net/t_wada/tdd-agile-samurai-base-camp

    View Slide

  27. σʔλΫϨϯδϯάϓϩάϥϜ
    ͷ։ൃαΠΫϧ
    ϚτϞͳσʔλΛߟ͑Δ
    ϚτϞͳσʔλΛzҙਤͨ͠ܗࣜzʹม׵ͨ͠
    ݁ՌΛςετέʔεͱͯ͠௥Ճ͢Δ
    ͦͷςετΛ࣮ߦࣦͯ͠ഊͤ͞Δ 3FE

    lҙਤͨ͠ܗࣜzʹม׵͢ΔίʔυΛॻ͘
    Ͱॻ͍ͨςετΛ੒ޭͤ͞Δ (SFFO

    ςετ͕௨Δ··ͰϦϑΝΫλϦϯάΛߦ͏
    3FGBDUPS

    ߥΕͨσʔλΛ௥Ճͯ͠ʙΛ܁Γฦ͢

    View Slide

  28. 28
    import (
    "testing"
    "github.com/stretchr/testify/assert”
    )
    func TestCleansingExample (t *testing.T) {
    var testCases := []struct {
    desc string
    in string
    want string
    }{
    {“ϚτϞͳσʔλ”,
    “03-1234-5678”,
    “03-1234-5678”},
    {“ϝΞυೖͬͯΔߥΕͨσʔλ”,
    “TEL:03-1234-9876 email:[email protected]”,
    “03-1234-9876”},
    }
    for _, test := range testCases {
    assert.Equal(t, Cleansing(test.in), test.want, test.desc)
    }
    }
    ͜͜ʹ
    lߥΕͨσʔλzͱ
    l๬·͍͠σʔλzͷ
    ςετέʔεΛՃ͑ͯɺ
    5%%ͷαΠΫϧΛճ͢

    View Slide

  29. 29
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    2017
    σʔλ
    ΫϨϯδϯά΋
    ͜ͷαΠΫϧΛ
    ճ͢

    View Slide

  30. ͜ͷಓͷΓͰσʔλΫϨϯδϯά
    ϓϩάϥϜΛҭͯͯʜ

    View Slide

  31. ෼ੳͰ͖Δɺ͖Ε͍ͳσʔλ΁

    ͜ͷಓͷΓͰߴ͍ਫ਼౓ͷ
    σʔληοτΛ࡞Δ

    View Slide

  32. !
    mecompany Inc. 32


    / -0
    / -0
    "
    201898
    લॲཧର৅σʔλ ԯ௒߲໨
    ʢ਺ेສϨίʔυ º਺ສ߲໨ʣ
    %'+.*

    +.*,.)
    ࠃ಺๏ਓ ສϨίʔυ
    ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλΛ౷߹͠
    l࢖͑Δzσʔλʹ͢ΔͨΊɺલॲཧ͕ඞཁ
    #(
    $&
    +.*,.)
    ଞ͔ࣾΒߪೖͨ͠σʔλϕʔε ਺ԯϨίʔυ
    σʔλ։ൃͷݱ৔

    View Slide

  33. 33
    େྔͷσʔλΫϨϯδϯάͰ
    ॲཧ݁ՌΛ֬ೝ͢ΔͨΊʹ
    • খ͍͞σʔληοτͰ
    ϑΟʔυόοΫαΠΫϧΛখ͘͞ճ͢
    • ΑΓେ͖͍σʔληοτΛ૬खʹ
    ໷ؒόονͰ$*Λճ͢
    • l໊دͤz΍ՄࢹԽγεςϜ·ͰσʔλΛ
    σϦόϦͯ͠ϑΟʔυόοΫαΠΫϧΛ
    ճ͢

    View Slide

  34. 201898 34
    σʔλΫϨϯδϯάΛ
    ͖ͬͪΓ΍ΔͱԿ͕ඒຯ͍͔͠
    • දهΏΕΛഉআͯ͠NFDBCͰ
    ෼͔ͪॻ͖͠΍͍͢σʔλʹͳΔ
    • ༨෼ͳจࣈྻΛഉআͯ͠ɺଞͷ
    σʔληοτͱܨ͛ΒΕΔ
    • ΧςΰϦ෼͚ϥϕϦϯά͕͖Ε͍ʹ
    Ͱ͖ͯɺσʔλ෼ੳਫ਼౓্͕͕Δ

    View Slide

  35. 201898 35
    ػցֶशͰ΋
    ਫ਼౓େࣄ
    Ͱ͢ΑͶ

    View Slide

  36. 36
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    σʔλͷ࡞ΓࠐΈɺػցֶशͷΠϯϑϥ͸
    ैདྷͷΤϯδχΞϦϯά͕౔୆

    View Slide

  37. 37
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    ػցֶशϞσϧ΋ɺσʔλ΋
    ϑΟʔυόοΫΛճͯ͠ҭͯΔ

    View Slide

  38. 38
    https://www.slideshare.net/hironoriwashizaki/ess2018-112258214
    ͜ͷ͋ͨΓͷ࿩͸
    lεϚʔτΤεΠʔzΛ
    ཤम͢Δͱ͍͍ͱࢥ͍·͢
    ʢࢲ͕ཤम͍ͨ͠ʣ

    View Slide

  39. mecompany Inc. 39
    ެతͳެ։৘ใ͔Β z࢖͑Δzσʔλʹ࢓্ཱͯ͛ͨ
    ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔε
    ϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ
    201898
    σʔλ։ൃͷݱ৔

    View Slide

  40. 40
    • σʔλͦͷ΋ͷͷ඼࣭Λ
    Ҿ্͖͛ΔͨΊʹ
    σʔλΫϨϯδϯάΛ͢Δ
    • σʔλΫϨϯδϯάΛ
    ͢ΔͨΊʹϓϩάϥϜΛॻ͘
    • σʔλΫϨϯδϯάϓϩάϥϜͷ
    ඼࣭Λ୲อ͢ΔͨΊʹ
    ςετίʔυ΋ॻ͘
    z࢖͑ΔzσʔλΛ࢓ཱͯΔͨΊʹ

    View Slide

  41. 41
    ۩ମతͳऔΓ૊Έ͸

    EC UFDITIPXDBTF
    %ηογϣϯ
    l೔ຊൃʂ.Z42-6%'ͱ(PݴޠͰ
    ࡞ΔϏοάσʔλલॲཧج൫z
    Ͱ͓࿩͠·͢
    https://www.db-tech-showcase.com/dbts/tokyo

    View Slide

  42. 42
    ۩ମతͳऔΓ૊Έ͸
    r
    $PEF GPS +BQBO 4VNNJU ৽ׁ
    lόουΦʔϓϯσʔλڙཆࣉz
    Ͱ͓࿩͠·͢
    https://summit2018.code4japan.org/session/210/
    ։ࣔՄೳͳόουσʔλͷ౤ߘ
    ͓଴͓ͪͯ͠Γ·͢

    View Slide

  43. 201898 43
    ݁ہͷͱ͜Ζ

    View Slide

  44. 201898 44
    ػցֶश͸
    ΞδϟΠϧ։ൃ
    ͕zલఏz

    View Slide

  45. 45
    5%%΋
    ͠ͳ͍Ͱ
    z"*z
    ͱ͸Կࣄͩ
    91ࡇΓ

    View Slide

  46. 201898 46
    lΞδϟΠϧ։ൃ
    ·ͩ΍ͬͯͳ͍z

    View Slide

  47. 201898 47
    ͋ͨͬͯ͠
    ͍·͞ο

    View Slide

  48. 48
    91ͱ͸
    ιʔγϟϧνΣϯδ
    Ͱ͋Δ
    UPCFDPOUJOVFˠ EC UFDITIPXDBTF

    View Slide