Collecting logs from mobile apps

Collecting logs from mobile apps

666ef10ec14e5a23d0fcf05bd2665575?s=128

rejasupotaro

April 15, 2016
Tweet

Transcript

  1. Collecting logs from mobile apps Kentaro Takiguchi

  2. About our service

  3. None
  4. None
  5. non-RTL RTL flipped

  6. Each user has a different context and background. Understanding users

    is important.
  7. Build - Measure - Learn

  8. Build - Measure - Learn

  9. None
  10. What is “fluentd”? “ ” Fluentd is an open source

    data collector, which lets us unify the data collection and consumption for a better use and understanding of data.
  11. … …

  12. … …

  13. … … ? retryable retryable retryable

  14. … … Filter / Buffer / Route

  15. <match td.*.*> @id td_wild type tdlog buffer_type file max_retry_wait 1h

    … </match> { tag: td.global.recipe_search_query_logs, timestamp: xxx, country: xxx, language: xxx, … } ```ruby Fluent::Logger::FluentLogger.new(tag_prefix, args).post(tag, map) ```
  16. ?

  17. Offline ? Connectivity Correctness Performance

  18. Data collector for mobile apps “Puree” https:/ /github.com/cookpad/puree-android https:/ /github.com/cookpad/puree-ios

  19. Unified logging layer for mobile apps Puree has following features

    Buffering Filtering Batching Retrying Pluggable
  20. Search “chicken” public class RecipeSearchLog implements PureeLog { @SerializedName("event") String

    event; @SerializedName("keyword") String keyword; @SerializedName("order") String order; @SerializedName("page") int page; @SerializedName("per_page") int perPage; @SerializedName("total_hits") int totalHits; } Puree.send(new RecipeSearchLog(…)); {“event”:”recipe.search”, “keyword”:”chicken”,…} Track events
  21. Buffering & Batching log logs {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”,

    “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ]
  22. Retrying log logs …

  23. Filtering drop SamplingFilter TimestampFilter add timestamp

  24. Pluggable output plugins • custom filters • flush interval millis

    • logs per request • max retry count
  25. Find a bottleneck using Puree ? Image download is so

    slow!
  26. image size image format network type S3 fetch time image

    convert time CDN cache hit ratio local cache hit ratio decode time render time number of threads device’s CPU / RAM OS version ?
  27. { "event": "image.download", "cache": "TCP_MISS", "cache_remote": "TCP_HIT", "content_length": "17670", "content_type":

    "image/webp", "convert_time_ms": "64.445", "network_type": "4G", "s3_fetch_time_ms": "41.575", "total_time_ms": 1471, … } SELECT cache, network_type, ROUND(AVG(total_time_ms), 0) AS total_time_ms FROM image_download_logs WHERE TD_TIME_RANGE(…) GROUP BY cache, network_type SamplingFilter (1%) TimestampFilter
  28. Collecting logs from mobile apps {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”,

    “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ] add user_id, country, … {“tag”:”td.global.recipe_search”, “user_id”:”xxx”, …} SELECT * FROM table WHERE …