Collecting logs from mobile apps

Transcript

1. Collecting logs from mobile apps
   Kentaro Takiguchi
   

   View Slide

2. About our service
   

   View Slide

3. View Slide

4. View Slide

5. non-RTL
   RTL
   flipped
   

   View Slide

6. Each user has a different context and background.
   Understanding users is important.
   

   View Slide

7. Build - Measure - Learn
   

   View Slide

8. Build - Measure - Learn
   

   View Slide

9. View Slide

10. What is “fluentd”?
    “ ”
    Fluentd is an open source data collector, which lets us unify the data
    collection and consumption for a better use and understanding of data.
    

    View Slide

11. … …
    

    View Slide

12. … …
    

    View Slide

13. … …
    ?
    retryable
    retryable
    retryable
    

    View Slide

14. … …
    Filter / Buffer / Route
    

    View Slide

15. 
    @id td_wild
    type tdlog
    buffer_type file
    max_retry_wait 1h
    …
    
    {
    tag: td.global.recipe_search_query_logs,
    timestamp: xxx,
    country: xxx,
    language: xxx,
    …
    }
    ```ruby
    Fluent::Logger::FluentLogger.new(tag_prefix, args).post(tag, map)
    ```
    

    View Slide

16. ?
    

    View Slide

17. Offline
    ?
    Connectivity
    Correctness
    Performance
    

    View Slide

18. Data collector for mobile apps
    “Puree”
    https:/
    /github.com/cookpad/puree-android
    https:/
    /github.com/cookpad/puree-ios
    

    View Slide

19. Unified logging layer for mobile apps
    Puree has following features
    Buffering
    Filtering
    Batching
    Retrying
    Pluggable
    

    View Slide

20. Search “chicken”
    public class RecipeSearchLog implements PureeLog {
    @SerializedName("event") String event;
    @SerializedName("keyword") String keyword;
    @SerializedName("order") String order;
    @SerializedName("page") int page;
    @SerializedName("per_page") int perPage;
    @SerializedName("total_hits") int totalHits;
    }
    Puree.send(new RecipeSearchLog(…));
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    Track events
    

    View Slide

21. Buffering & Batching
    log
    logs
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    [
    {“event”:”recipe.search”, “keyword”:”chicken”,…},
    {“event”:”recipe.search”, “keyword”:”chicken”,…},
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    ]
    

    View Slide

22. Retrying
    log
    logs
    …
    

    View Slide

23. Filtering
    drop
    SamplingFilter TimestampFilter
    add timestamp
    

    View Slide

24. Pluggable
    output plugins
    • custom filters
    • flush interval millis
    • logs per request
    • max retry count
    

    View Slide

25. Find a bottleneck using Puree
    ?
    Image download is so slow!
    

    View Slide

26. image size
    image format
    network type
    S3 fetch time
    image convert time
    CDN cache hit ratio
    local cache hit ratio
    decode time
    render time
    number of threads
    device’s CPU / RAM
    OS version
    ?
    

    View Slide

27. 
    {
    "event": "image.download",
    "cache": "TCP_MISS",
    "cache_remote": "TCP_HIT",
    "content_length": "17670",
    "content_type": "image/webp",
    "convert_time_ms": "64.445",
    "network_type": "4G",
    "s3_fetch_time_ms": "41.575",
    "total_time_ms": 1471,
    …
    }
    SELECT
    cache,
    network_type,
    ROUND(AVG(total_time_ms), 0) AS total_time_ms
    FROM
    image_download_logs
    WHERE
    TD_TIME_RANGE(…)
    GROUP BY
    cache, network_type
    SamplingFilter (1%)
    TimestampFilter
    

    View Slide

28. Collecting logs from mobile apps
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    [
    {“event”:”recipe.search”, “keyword”:”chicken”,…},
    {“event”:”recipe.search”, “keyword”:”chicken”,…},
    {“event”:”recipe.search”, “keyword”:”chicken”,…}
    ]
    add user_id, country, …
    {“tag”:”td.global.recipe_search”, “user_id”:”xxx”, …}
    SELECT * FROM table WHERE …
    
    

    View Slide