Slide 1

Slide 1 text

Why and How SmartNews uses SaaS? @takus 2015.06.26 SmartNews Official Character Chikyu-Kun

Slide 2

Slide 2 text

Takumi Sakamoto (@takus)

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

SmartNews Crosses 10 Million Downloads in Global Source: App Annie

Slide 5

Slide 5 text

SmartNews has No.1 MAU in News Apps, Japan Comparison of the other News Apps 4.3M MAU Source: Nielsen Japan

Slide 6

Slide 6 text

Ranking of Active Users  * 2 5 1 1 199     -‐‑‒ E . 1 2 3 4 5 6 7 8 9 10 1 B 1 2 A 0 1 5

Slide 7

Slide 7 text

Long Time Spent Facebook messenger LINE Facebook Gmail Youtube COOKPAD News App A SmartNews News App B News App C ※ Spending minutes/person, month Source: Nielsen Japan

Slide 8

Slide 8 text

Why SaaS? Use cases Relationship Summary

Slide 9

Slide 9 text

We want to focus on OUR PRODUCT

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

We must move fast

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Engineering resources are limited

Slide 14

Slide 14 text

Efficient Ads System Document Classification Site Reliability Engineering High Performance API Server News Crawler Realtime Ranking Algorithm Data Management Platform UI/UX

Slide 15

Slide 15 text

Can we assign excellent engineers to every positions?

Slide 16

Slide 16 text

No, probably

Slide 17

Slide 17 text

The answer is

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Why SaaS? • Solves specific issues for us • monitoring, alerting, visualising etc... • Less bootstrap process • tiny setup process (a java agent, a chef recipe etc...) • built-in user friendly web Interface • No dedicated person is needed to develop/operate • experts in an area do it well • easy to manage with programable API

Slide 20

Slide 20 text

Why SaaS? Use cases Relationship Summary

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

NewRelic • (Application) Monitoring as a Service • easy setup for popular language (e.g. Java) • add method traces by annotation • trace SQL queries automatically • records deployments

Slide 23

Slide 23 text

Setup for Tomcat • deploy newrelic archives to $CATALINA_HOME • edit newrelic.yml • application name • add -javaagent • $CATALINA_HOME/newrelic/newrelic.jar • add environment if necessary • -Dnewrelic.environment=xxx

Slide 24

Slide 24 text

Customize for each Apps # add method tracer @Trace(metricName="API/yourMethod", dispatcher=true) public Response yourMethod() { ... ... ... } # add custom metric MetricAggregator aggregator = NewRelic.getAgent().getMetricAggregator(); aggregator.recordMetric(key, value); # record exception NewRelic.noticeError(ex);

Slide 25

Slide 25 text

Trace methods & queries

Slide 26

Slide 26 text

Record Deployments

Slide 27

Slide 27 text

from fabric.api run from fabric.decorators import runs_once, parallel @parallel @task def deploy(): do_your_task() notify() @runs_once def notify(): ver = release_version() run("java -jar newrelic.jar deployment --appname=xxxx --revision=%s" % ver) Record deployment by fabric https://docs.newrelic.com/docs/agents/java-agent/instrumentation/recording-deployments-java-agent

Slide 28

Slide 28 text

NewRelic Insights Query to custom metrics via NRQL SmartNews seems top 10 user of Insights

Slide 29

Slide 29 text

What's great? • Developer can monitor their own app by themselves • adding custom tracer, custom metrics • Don't have to prepare profiler for each language • Java, Scala, Ruby, (Golang), etc... • suit for Microservices

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Datadog • (System) Monitoring as a Service • easy to setup with Chef • 100+ built in integration (AWS, MySQL, Docker...) • correlate metrics with events • tag-based flexible visualization • build your own interactive dashboards

Slide 32

Slide 32 text

Setup dd-agent with Chef # Install cookbook knife cookbook site install datadog # Set Datadog-specific attributes node.default['datadog']['api_key'] = "xxxxxxxxxx" # Add a recipe in your run list "run_list": ["recipe[datadog::dd-agent]"]

Slide 33

Slide 33 text

BTW, What is dd-agent? http://help.datadoghq.com/hc/en-us/articles/203034929-What-is-the-Datadog-Agent-What-resources-does-it-consume-

Slide 34

Slide 34 text

Built-in Integrations

Slide 35

Slide 35 text

Setup MySQL Monitor # Edit conf.d/mysql.yaml instances: - server: localhost user: datadog pass: xxxxxxxx tags: - dbtype:master options: replication: 0

Slide 36

Slide 36 text

Correlate metrics w/ deployment

Slide 37

Slide 37 text

New Relic Integration

Slide 38

Slide 38 text

sum:apache.status_2xx_count{role:web} by {env} EC2 Classic VPC Tag-based flexible visualization

Slide 39

Slide 39 text

http://qiita.com/takus/items/c1d71bfcc231d0c24e0a

Slide 40

Slide 40 text

Migrate from EC2 Classic to VPC with Route53

Slide 41

Slide 41 text

role = web env = classic role = web env = vpc Provision clusters with Chef

Slide 42

Slide 42 text

role = web env = classic role = web env = vpc Aggregate metrics with 'env' tag

Slide 43

Slide 43 text

Check clusters by Host Maps https://www.datadoghq.com/blog/2015/03/introducing-host-maps-know-thy-infrastructure/

Slide 44

Slide 44 text

Build your own dashboards

Slide 45

Slide 45 text

PagerDuty Integration Only pager for related teams (members)

Slide 46

Slide 46 text

What's great? • No dedicated Ops required for system monitoring • chef cookbook & built-in integration • customisable dashboard for each developers • Find the root cause quickly • gather all of metrics/events/alerting at one place • tag-based flexible visualization • correlating metrics & events

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Chartio • Dashboard as a Service • create beautiful dashboards • provide connectors to various data sources • MySQL, Redshift, Google Analytics, BigQuery, etc... • Presto with Prestogres • non-engineer can create dashboard w/o SQL

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

How to create charts? 1. Build a query to data source (drag & drop / SQL) 2. Modify data (filterɺsortɺadd column) 3. Visualise it (table, pie chart, bar chart)

Slide 51

Slide 51 text

Connecting to Presto https://github.com/treasure-data/prestogres http://docs.treasuredata.com/articles/chartio

Slide 52

Slide 52 text

Join BigQuery & MySQL Layer1 BigQuery Result article_id country pv 1 JP 1000000 2 US 2000000 3 GB 3000000 4 JP 4000000 article_id url title 1 http://aaa Article A 2 http://bbb Article B 3 http://ccc Article C 4 http://ddd Article D Layer2 MySQL Result Join https://support.chartio.com/docs/charts/#merging-data-joins

Slide 53

Slide 53 text

Join BigQuery & MySQL article_id country pv url title 1 JP 1000000 http://aaa Article A 2 US 2000000 http://bbb Article B 3 GB 3000000 http://ccc Article C 4 JP 4000000 http://ddd Article D Joined Layer : BigQuery + MySQL https://support.chartio.com/docs/charts/#merging-data-joins CSV Graph Mail

Slide 54

Slide 54 text

Categorical Drop down Could you give me the ios's article ranking of bbc.co.uk in UK? e.g. : A request from BizDev team

Slide 55

Slide 55 text

How is the result of a A/B test in California? It seems that iPhone 6's user complaints about new features. Can I compare with the result of each devices? e.g. : A request from client developers Categorical Drop down

Slide 56

Slide 56 text

What's great? • No dedicated SQL Monkey required for visualising • everyone can be SQL Monkey • non-engineer can do by WebUI • Easy to drill down from overview to each users • overall summaries • each OS, country, region, language, etc...

Slide 57

Slide 57 text

c.f. About Our Data Platform https://goo.gl/Rc1uuO http://goo.gl/KL1fpL

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

OneLogin • Identity as a Service • provides single sign on to various SaaS • SAML / Auto complete form with browser extension • non-frustrated 2FA with mobile app

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

Why SaaS? Use cases Relationship Summary

Slide 63

Slide 63 text

Send a Pull Request

Slide 64

Slide 64 text

Send a Pull Request • Wrote a fluentd - datadog integration • only aiming for meeting our requirements • It became an official integration • https://www.datadoghq.com/2015/02/monitor- fluentd-datadog/

Slide 65

Slide 65 text

Hold an User Meetup http://blog.takus.me/2015/02/26/dd-sushi/

Slide 66

Slide 66 text

Gifts from SaaS

Slide 67

Slide 67 text

Discuss products in conferences

Slide 68

Slide 68 text

Discuss products in conferences • Fortunately, I had some chances to attend them • Velocity Conference 2013, 2015 • Amazon re:Invent 2014 • Great time to talk with core developers • learn new services & new features • explain our feature requests by myself • ask their product roadmap

Slide 69

Slide 69 text

New Relic's Service Map

Slide 70

Slide 70 text

Anomaly detection by SumoLogic

Slide 71

Slide 71 text

Future of SaaS • SaaS is becoming hot area in startup • 2014/12/11: NewRelic raised $115M in IPO • 2015/01/29: Datadog raised $31M in Funding • 2015/06/01: SumoLogic raised $80M in Funding • SaaS will evolve rapidly with a large amount of funds? • probably in my opinion

Slide 72

Slide 72 text

Why SaaS? Use cases Relationship Summary

Slide 73

Slide 73 text

SaaS makes life easier • The real world is complex. We have a lot of tasks to do • Professionals solve specific issues in behalf of us • No dedicated engineer is needed for it • We can focus on our own problems

Slide 74

Slide 74 text

We’re hiring! Site reliability engineer iOS/Android engineer Web application engineer Productivity engineer ML/NLP engineer Growth hack engineer Ad engineer http://about.smartnews.com/en/careers/

Slide 75

Slide 75 text

One more thing...

Slide 76

Slide 76 text

Join Our Free Lunch! We love talking about Technologies, Products, Medias, etc...