$30 off During Our Annual Pro Sale. View Details »

[PyConKR2017] Best-first search를 이용한 다차원 큐브 탐색

[PyConKR2017] Best-first search를 이용한 다차원 큐브 탐색

Jangsea Park

August 12, 2017
Tweet

More Decks by Jangsea Park

Other Decks in Programming

Transcript

  1. Best-first searchܳ ੉ਊೠ
    ׮ରਗ ௸࠳ ఐ࢝
    -[|]- Box and Whisker
    ߅੢द

    View Slide

  2. Commerce platform
    Data, Google Analytics…

    View Slide

  3. Google Analytics
    powerful but ...

    View Slide

  4. Dimension + Metrics

    View Slide

  5. Dims’ Combination + Metrics

    View Slide

  6. Many dimensions
    https://developers.google.com/analytics/devguides/reporting/core/dimsmets
    200+

    View Slide

  7. More combinations
    137,979

    View Slide

  8. Much more numbers
    (Region, Gender)
    #32 lines(segments)

    View Slide

  9. Much more numbers
    (A,) = 5
    (A, B) = 25
    (A, B, C) = 125

    (A, B, …, G) = 5 ** 7
    6.7B

    View Slide

  10. Multidimensional cube
    s: 427
    r: 87
    d: 122
    s: 283
    r: 23
    d: 124
    s: 291
    r: 19
    d: 872
    s: 123
    r: 108
    d: 503
    s: 287
    r: 13
    d: 687
    s: 987
    r: 432
    d: 1113
    s: 293
    r: 18
    d: 983
    s: 293
    r: 2929
    d: 7023
    s: 111
    r: 72
    d: 982
    s: 293
    r: 199
    d: 777
    s: 28
    r: 1
    d: 134
    s: 19
    r: 0
    d: 29
    s: 100
    r: 29
    d: 89
    s: 192
    r: 19
    d: 192
    s: 293
    r: 18
    d: 298
    s: 19
    r: 1
    d: 29
    Region
    Gender
    Age
    The most efficient

    View Slide

  11. GA Reporting API
    google-api-python-client

    View Slide

  12. Restrictions
    137,979
    10 queries per second (QPS) per IP address
    10 concurrent requests per view (profile)
    137,979 / 10 / 60 = 230min
    10,000 requests per view (profile) per day

    View Slide

  13. Better way
    Gender Age Region
    /
    Campaign Usertype
    Gender,
    Region
    Age,
    Campaign
    Region,
    Usertype
    Region,
    Usertype,
    Gender
    Region,
    Usertype,
    Campaign
    Usertype,
    Age
    Region,
    Usertype,
    Gender,
    Age
    s: 427
    r: 87
    d: 122
    s: 283
    r: 23
    d: 124
    s: 291
    r: 19
    d: 872
    s: 123
    r: 108
    d: 503
    s: 287
    r: 13
    d: 687
    s: 987
    r: 432
    d: 1113
    s: 293
    r: 18
    d: 983
    s: 293
    r: 2929
    d: 7023
    s: 111
    r: 72
    d: 982
    s: 293
    r: 199
    d: 777
    s: 28
    r: 1
    d: 134
    s: 19
    r: 0
    d: 29
    s: 100
    r: 29
    d: 89
    s: 192
    r: 19
    d: 192
    s: 293
    r: 18
    d: 298
    s: 19
    r: 1
    d: 29
    Region
    Gender
    Age

    View Slide

  14. Graph search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O

    View Slide

  15. Graph search

    View Slide

  16. Breadth-first search

    View Slide

  17. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:

    View Slide

  18. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: B
    C

    View Slide

  19. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: C
    D
    E
    B

    View Slide

  20. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: D
    E
    B C
    F
    G

    View Slide

  21. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D
    E
    B C
    F
    G
    H
    I

    View Slide

  22. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C
    F
    G
    H
    I
    J
    K

    View Slide

  23. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C F
    G
    H
    I
    J
    K
    L

    View Slide

  24. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C F
    G
    H
    I
    J
    K
    L

    View Slide

  25. Depth-first search

    View Slide

  26. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:

    View Slide

  27. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: B C

    View Slide

  28. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D

    View Slide

  29. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D
    I
    H

    View Slide

  30. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D
    I
    H

    View Slide

  31. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H

    View Slide

  32. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H
    K
    J

    View Slide

  33. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H
    K
    J

    View Slide

  34. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H K
    J

    View Slide

  35. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J
    F G

    View Slide

  36. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J F
    G

    View Slide

  37. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J F
    G

    View Slide

  38. Best-first search

    View Slide

  39. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6

    View Slide

  40. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    5+10=15
    4+12=16

    View Slide

  41. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    4+12=16
    12+7=19
    11+5=16

    View Slide

  42. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    8+6=14
    12+7=19
    11+5=16

    View Slide

  43. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    11+4=15
    12+7=19
    10+5=15

    View Slide

  44. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    11+4=15
    12+7=19
    12+4=16

    View Slide

  45. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18 13+4=17
    12+7=19
    12+4=16
    15+0=15

    View Slide

  46. More about Best-first search
    heapq — Heap queue algorithm

    https://docs.python.org/3.6/library/heapq.html#theory
    [AIMA] 3.5 INFORMED (HEURISTIC) SEARCH STRATEGIES
    [AIMA] 3.6 HEURISTIC FUNCTIONS

    View Slide

  47. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    S: 1000
    R: 10000

    View Slide

  48. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    S: 1000
    R: 10000
    Female
    S: 600
    R: 8000
    Male
    S: 400
    R: 2000

    View Slide

  49. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type

    View Slide

  50. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    1.0 1.0 1.0 1.0
    0.8
    0.8
    0.8 0.8 0.8

    View Slide

  51. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    0.6 1.0 1.0 1.0
    0.8
    10s
    S: 100
    R: 200
    20s
    S: 500
    R: 2000
    30s
    S: 200
    R: 6000
    40s
    S: 100
    R: 1000
    50+
    S: 100
    R: 800

    View Slide

  52. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 1.0 1.0 1.0
    0.48
    0.8
    0.8 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.6 0.6 0.6

    View Slide

  53. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 1.0 1.0
    0.48
    0.8
    0.32 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.6 0.6

    View Slide

  54. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 1.0 1.0
    0.48
    0.8
    0.32 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.6 0.6
    Region
    Camp
    Region
    Type
    0.4 0.4

    View Slide

  55. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 1.0
    0.48
    0.8
    0.32 0.16 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.6
    Region
    Camp
    Region
    Type
    0.08 0.4
    Camp
    Type
    0.2

    View Slide

  56. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 0.5
    0.48
    0.8
    0.32 0.16 0.4
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.3
    Region
    Camp
    Region
    Type
    0.08 0.2
    Camp
    Type
    0.1

    View Slide

  57. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 0.5
    0.48
    0.8
    0.32 0.16 0.4
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.3
    Region
    Camp
    Region
    Type
    0.08 0.2
    Camp
    Type
    0.1

    View Slide

  58. Report

    View Slide

  59. Report

    View Slide

  60. Efficiency

    View Slide

  61. Where to dig

    View Slide

  62. Q&A

    View Slide