[PyConKR2017] Best-first search를 이용한 다차원 큐브 탐색

Transcript

1. Best-first searchܳ ੉ਊೠ
   ׮ରਗ ௸࠳ ఐ࢝
   -[|]- Box and Whisker
   ߅੢द
   

   View Slide

2. Commerce platform
   Data, Google Analytics…
   

   View Slide

3. Google Analytics
   powerful but ...
   

   View Slide

4. Dimension + Metrics
   

   View Slide

5. Dims’ Combination + Metrics
   

   View Slide

6. Many dimensions
   https://developers.google.com/analytics/devguides/reporting/core/dimsmets
   200+
   

   View Slide

7. More combinations
   137,979
   

   View Slide

8. Much more numbers
   (Region, Gender)
   #32 lines(segments)
   

   View Slide

9. Much more numbers
   (A,) = 5
   (A, B) = 25
   (A, B, C) = 125
   …
   (A, B, …, G) = 5 ** 7
   6.7B
   

   View Slide

10. Multidimensional cube
    s: 427
    r: 87
    d: 122
    s: 283
    r: 23
    d: 124
    s: 291
    r: 19
    d: 872
    s: 123
    r: 108
    d: 503
    s: 287
    r: 13
    d: 687
    s: 987
    r: 432
    d: 1113
    s: 293
    r: 18
    d: 983
    s: 293
    r: 2929
    d: 7023
    s: 111
    r: 72
    d: 982
    s: 293
    r: 199
    d: 777
    s: 28
    r: 1
    d: 134
    s: 19
    r: 0
    d: 29
    s: 100
    r: 29
    d: 89
    s: 192
    r: 19
    d: 192
    s: 293
    r: 18
    d: 298
    s: 19
    r: 1
    d: 29
    Region
    Gender
    Age
    The most efficient
    

    View Slide

11. GA Reporting API
    google-api-python-client
    

    View Slide

12. Restrictions
    137,979
    10 queries per second (QPS) per IP address
    10 concurrent requests per view (profile)
    137,979 / 10 / 60 = 230min
    10,000 requests per view (profile) per day
    

    View Slide

13. Better way
    Gender Age Region
    /
    Campaign Usertype
    Gender,
    Region
    Age,
    Campaign
    Region,
    Usertype
    Region,
    Usertype,
    Gender
    Region,
    Usertype,
    Campaign
    Usertype,
    Age
    Region,
    Usertype,
    Gender,
    Age
    s: 427
    r: 87
    d: 122
    s: 283
    r: 23
    d: 124
    s: 291
    r: 19
    d: 872
    s: 123
    r: 108
    d: 503
    s: 287
    r: 13
    d: 687
    s: 987
    r: 432
    d: 1113
    s: 293
    r: 18
    d: 983
    s: 293
    r: 2929
    d: 7023
    s: 111
    r: 72
    d: 982
    s: 293
    r: 199
    d: 777
    s: 28
    r: 1
    d: 134
    s: 19
    r: 0
    d: 29
    s: 100
    r: 29
    d: 89
    s: 192
    r: 19
    d: 192
    s: 293
    r: 18
    d: 298
    s: 19
    r: 1
    d: 29
    Region
    Gender
    Age
    

    View Slide

14. Graph search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    

    View Slide

15. Graph search
    

    View Slide

16. Breadth-first search
    

    View Slide

17. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    

    View Slide

18. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: B
    C
    

    View Slide

19. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: C
    D
    E
    B
    

    View Slide

20. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: D
    E
    B C
    F
    G
    

    View Slide

21. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D
    E
    B C
    F
    G
    H
    I
    

    View Slide

22. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C
    F
    G
    H
    I
    J
    K
    

    View Slide

23. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C F
    G
    H
    I
    J
    K
    L
    

    View Slide

24. Breadth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    D E
    B C F
    G
    H
    I
    J
    K
    L
    

    View Slide

25. Depth-first search
    

    View Slide

26. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    

    View Slide

27. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier: B C
    

    View Slide

28. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D
    

    View Slide

29. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D
    I
    H
    

    View Slide

30. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D
    I
    H
    

    View Slide

31. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H
    

    View Slide

32. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H
    K
    J
    

    View Slide

33. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H
    K
    J
    

    View Slide

34. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B
    C
    E
    D I
    H K
    J
    

    View Slide

35. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J
    F G
    

    View Slide

36. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J F
    G
    

    View Slide

37. Depth-first search
    A
    B
    D
    H I
    E
    J K
    C
    F
    L M
    G
    N O
    A
    Explored:
    Frontier:
    B C
    E
    D I
    H K
    J F
    G
    

    View Slide

38. Best-first search
    

    View Slide

39. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    

    View Slide

40. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    5+10=15
    4+12=16
    

    View Slide

41. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    4+12=16
    12+7=19
    11+5=16
    

    View Slide

42. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    8+6=14
    12+7=19
    11+5=16
    

    View Slide

43. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    11+4=15
    12+7=19
    10+5=15
    

    View Slide

44. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18
    11+4=15
    12+7=19
    12+4=16
    

    View Slide

45. Best-first search
    A
    B C
    D
    E
    F
    G H
    I
    J
    K
    L
    M
    N
    O
    P
    3
    4
    5
    3
    2
    4
    7
    6
    4
    3
    2
    1
    2
    2
    3
    4
    1
    2
    4
    A: 15
    B: 15
    C: 10
    D: 12
    E: 12
    F: 7
    G: 11
    H: 5
    I: 5
    J: 6
    K: 4
    L: 4
    M: 0
    N: 1
    O: 4
    P: 6
    3+15=18 13+4=17
    12+7=19
    12+4=16
    15+0=15
    

    View Slide

46. More about Best-first search
    heapq — Heap queue algorithm
    
    https://docs.python.org/3.6/library/heapq.html#theory
    [AIMA] 3.5 INFORMED (HEURISTIC) SEARCH STRATEGIES
    [AIMA] 3.6 HEURISTIC FUNCTIONS
    

    View Slide

47. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    S: 1000
    R: 10000
    

    View Slide

48. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    S: 1000
    R: 10000
    Female
    S: 600
    R: 8000
    Male
    S: 400
    R: 2000
    

    View Slide

49. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    

    View Slide

50. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    1.0 1.0 1.0 1.0
    0.8
    0.8
    0.8 0.8 0.8
    

    View Slide

51. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    0.6 1.0 1.0 1.0
    0.8
    10s
    S: 100
    R: 200
    20s
    S: 500
    R: 2000
    30s
    S: 200
    R: 6000
    40s
    S: 100
    R: 1000
    50+
    S: 100
    R: 800
    

    View Slide

52. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 1.0 1.0 1.0
    0.48
    0.8
    0.8 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.6 0.6 0.6
    

    View Slide

53. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 1.0 1.0
    0.48
    0.8
    0.32 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.6 0.6
    

    View Slide

54. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 1.0 1.0
    0.48
    0.8
    0.32 0.8 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.6 0.6
    Region
    Camp
    Region
    Type
    0.4 0.4
    

    View Slide

55. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 1.0
    0.48
    0.8
    0.32 0.16 0.8
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.6
    Region
    Camp
    Region
    Type
    0.08 0.4
    Camp
    Type
    0.2
    

    View Slide

56. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 0.5
    0.48
    0.8
    0.32 0.16 0.4
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.3
    Region
    Camp
    Region
    Type
    0.08 0.2
    Camp
    Type
    0.1
    

    View Slide

57. Best-first search for GA API
    Gender Age Region
    /
    Camp Type
    Gender
    Age
    Gender
    Region
    Gender
    Camp
    Gender
    Type
    0.6 0.4 0.2 0.5
    0.48
    0.8
    0.32 0.16 0.4
    Age
    Region
    Age
    Camp
    Age
    Type
    0.24 0.12 0.3
    Region
    Camp
    Region
    Type
    0.08 0.2
    Camp
    Type
    0.1
    

    View Slide

58. Report
    

    View Slide

59. Report
    

    View Slide

60. Efficiency
    

    View Slide

61. Where to dig
    

    View Slide

62. Q&A
    

    View Slide