Collecting logs from mobile apps

1. Collecting logs from mobile apps Kentaro Takiguchi
2. About our service
5. non-RTL RTL flipped
6. Each user has a different context and background. Understanding users is important.
7. Build - Measure - Learn
8. Build - Measure - Learn
10. What is “fluentd”? “ ” Fluentd is an open source data collector, which lets us unify the data collection and consumption for a better use and understanding of data.
11. … …
12. … …
13. … … ? retryable retryable retryable
14. … … Filter / Buffer / Route
15. <match td.*.*> @id td_wild type tdlog buffer_type file max_retry_wait 1h … </match> { tag: td.global.recipe_search_query_logs, timestamp: xxx, country: xxx, language: xxx, … } ```ruby Fluent::Logger::FluentLogger.new(tag_prefix, args).post(tag, map) ```
16. ?
17. Offline ? Connectivity Correctness Performance
18. Data collector for mobile apps “Puree” https:/ /github.com/cookpad/puree-android https:/ /github.com/cookpad/puree-ios
19. Unified logging layer for mobile apps Puree has following features Buffering Filtering Batching Retrying Pluggable
20. Search “chicken” public class RecipeSearchLog implements PureeLog { @SerializedName("event") String event; @SerializedName("keyword") String keyword; @SerializedName("order") String order; @SerializedName("page") int page; @SerializedName("per_page") int perPage; @SerializedName("total_hits") int totalHits; } Puree.send(new RecipeSearchLog(…)); {“event”:”recipe.search”, “keyword”:”chicken”,…} Track events
21. Buffering & Batching log logs {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ]
22. Retrying log logs …
23. Filtering drop SamplingFilter TimestampFilter add timestamp
24. Pluggable output plugins • custom filters • flush interval millis • logs per request • max retry count
25. Find a bottleneck using Puree ? Image download is so slow!
26. image size image format network type S3 fetch time image convert time CDN cache hit ratio local cache hit ratio decode time render time number of threads device’s CPU / RAM OS version ?
27. { "event": "image.download", "cache": "TCP_MISS", "cache_remote": "TCP_HIT", "content_length": "17670", "content_type": "image/webp", "convert_time_ms": "64.445", "network_type": "4G", "s3_fetch_time_ms": "41.575", "total_time_ms": 1471, … } SELECT cache, network_type, ROUND(AVG(total_time_ms), 0) AS total_time_ms FROM image_download_logs WHERE TD_TIME_RANGE(…) GROUP BY cache, network_type SamplingFilter (1%) TimestampFilter
28. Collecting logs from mobile apps {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ] add user_id, country, … {“tag”:”td.global.recipe_search”, “user_id”:”xxx”, …} SELECT * FROM table WHERE …