Collecting logs from mobile apps

Slide 1

Slide 1 text

Collecting logs from mobile apps Kentaro Takiguchi

Slide 2

Slide 2 text

About our service

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

non-RTL RTL ﬂipped

Slide 6

Slide 6 text

Each user has a different context and background. Understanding users is important.

Slide 7

Slide 7 text

Build - Measure - Learn

Slide 8

Slide 8 text

Build - Measure - Learn

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

What is “ﬂuentd”? “ ” Fluentd is an open source data collector, which lets us unify the data collection and consumption for a better use and understanding of data.

Slide 11

Slide 11 text

… …

Slide 12

Slide 12 text

… …

Slide 13

Slide 13 text

… … ? retryable retryable retryable

Slide 14

Slide 14 text

… … Filter / Buffer / Route

Slide 15

Slide 15 text

@id td_wild type tdlog buffer_type ﬁle max_retry_wait 1h … { tag: td.global.recipe_search_query_logs, timestamp: xxx, country: xxx, language: xxx, … } ```ruby Fluent::Logger::FluentLogger.new(tag_prefix, args).post(tag, map) ```

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Ofﬂine ? Connectivity Correctness Performance

Slide 18

Slide 18 text

Data collector for mobile apps “Puree” https:/ /github.com/cookpad/puree-android https:/ /github.com/cookpad/puree-ios

Slide 19

Slide 19 text

Uniﬁed logging layer for mobile apps Puree has following features Buffering Filtering Batching Retrying Pluggable

Slide 20

Slide 20 text

Search “chicken” public class RecipeSearchLog implements PureeLog { @SerializedName("event") String event; @SerializedName("keyword") String keyword; @SerializedName("order") String order; @SerializedName("page") int page; @SerializedName("per_page") int perPage; @SerializedName("total_hits") int totalHits; } Puree.send(new RecipeSearchLog(…)); {“event”:”recipe.search”, “keyword”:”chicken”,…} Track events

Slide 21

Slide 21 text

Buffering & Batching log logs {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ]

Slide 22

Slide 22 text

Retrying log logs …

Slide 23

Slide 23 text

Filtering drop SamplingFilter TimestampFilter add timestamp

Slide 24

Slide 24 text

Pluggable output plugins • custom ﬁlters • ﬂush interval millis • logs per request • max retry count

Slide 25

Slide 25 text

Find a bottleneck using Puree ? Image download is so slow!

Slide 26

Slide 26 text

image size image format network type S3 fetch time image convert time CDN cache hit ratio local cache hit ratio decode time render time number of threads device’s CPU / RAM OS version ?

Slide 27

Slide 27 text

{ "event": "image.download", "cache": "TCP_MISS", "cache_remote": "TCP_HIT", "content_length": "17670", "content_type": "image/webp", "convert_time_ms": "64.445", "network_type": "4G", "s3_fetch_time_ms": "41.575", "total_time_ms": 1471, … } SELECT cache, network_type, ROUND(AVG(total_time_ms), 0) AS total_time_ms FROM image_download_logs WHERE TD_TIME_RANGE(…) GROUP BY cache, network_type SamplingFilter (1%) TimestampFilter

Slide 28

Slide 28 text

Collecting logs from mobile apps {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ] add user_id, country, … {“tag”:”td.global.recipe_search”, “user_id”:”xxx”, …} SELECT * FROM table WHERE …