Why and How
SmartNews uses SaaS?
@takus
2015.06.26
SmartNews Official Character
Chikyu-Kun
Slide 2
Slide 2 text
Takumi Sakamoto
(@takus)
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
SmartNews Crosses 10 Million Downloads in Global
Source: App Annie
Slide 5
Slide 5 text
SmartNews has No.1 MAU in News Apps, Japan
Comparison of the other News Apps
4.3M
MAU
Source: Nielsen Japan
Slide 6
Slide 6 text
Ranking of Active Users
* 2 5 1 1 199
-‐‑‒ E .
1
2
3
4
5
6
7
8
9
10
1 B 1
2 A 0 1 5
Slide 7
Slide 7 text
Long Time Spent
Facebook
messenger
LINE Facebook Gmail Youtube COOKPAD News App
A
SmartNews News App
B
News App
C
※ Spending minutes/person, month
Source: Nielsen Japan
Slide 8
Slide 8 text
Why SaaS?
Use cases
Relationship
Summary
Slide 9
Slide 9 text
We want to focus on
OUR PRODUCT
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
We must move fast
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Engineering resources
are
limited
Slide 14
Slide 14 text
Efficient
Ads
System
Document
Classification
Site
Reliability
Engineering
High
Performance
API Server
News
Crawler
Realtime
Ranking
Algorithm
Data
Management
Platform
UI/UX
Slide 15
Slide 15 text
Can we assign
excellent engineers
to every positions?
Slide 16
Slide 16 text
No, probably
Slide 17
Slide 17 text
The answer is
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
Why SaaS?
• Solves specific issues for us
• monitoring, alerting, visualising etc...
• Less bootstrap process
• tiny setup process (a java agent, a chef recipe etc...)
• built-in user friendly web Interface
• No dedicated person is needed to develop/operate
• experts in an area do it well
• easy to manage with programable API
Slide 20
Slide 20 text
Why SaaS?
Use cases
Relationship
Summary
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
NewRelic
• (Application) Monitoring as a Service
• easy setup for popular language (e.g. Java)
• add method traces by annotation
• trace SQL queries automatically
• records deployments
Slide 23
Slide 23 text
Setup for Tomcat
• deploy newrelic archives to $CATALINA_HOME
• edit newrelic.yml
• application name
• add -javaagent
• $CATALINA_HOME/newrelic/newrelic.jar
• add environment if necessary
• -Dnewrelic.environment=xxx
Slide 24
Slide 24 text
Customize for each Apps
# add method tracer
@Trace(metricName="API/yourMethod", dispatcher=true)
public Response yourMethod() {
...
...
...
}
# add custom metric
MetricAggregator aggregator = NewRelic.getAgent().getMetricAggregator();
aggregator.recordMetric(key, value);
# record exception
NewRelic.noticeError(ex);
Slide 25
Slide 25 text
Trace methods & queries
Slide 26
Slide 26 text
Record Deployments
Slide 27
Slide 27 text
from fabric.api run
from fabric.decorators import runs_once, parallel
@parallel
@task
def deploy():
do_your_task()
notify()
@runs_once
def notify():
ver = release_version()
run("java -jar newrelic.jar deployment --appname=xxxx --revision=%s" % ver)
Record deployment by fabric
https://docs.newrelic.com/docs/agents/java-agent/instrumentation/recording-deployments-java-agent
Slide 28
Slide 28 text
NewRelic Insights
Query to custom metrics via NRQL
SmartNews seems top 10 user of Insights
Slide 29
Slide 29 text
What's great?
• Developer can monitor their own app by themselves
• adding custom tracer, custom metrics
• Don't have to prepare profiler for each language
• Java, Scala, Ruby, (Golang), etc...
• suit for Microservices
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
Datadog
• (System) Monitoring as a Service
• easy to setup with Chef
• 100+ built in integration (AWS, MySQL, Docker...)
• correlate metrics with events
• tag-based flexible visualization
• build your own interactive dashboards
Slide 32
Slide 32 text
Setup dd-agent with Chef
# Install cookbook
knife cookbook site install datadog
# Set Datadog-specific attributes
node.default['datadog']['api_key'] = "xxxxxxxxxx"
# Add a recipe in your run list
"run_list": ["recipe[datadog::dd-agent]"]
Slide 33
Slide 33 text
BTW, What is dd-agent?
http://help.datadoghq.com/hc/en-us/articles/203034929-What-is-the-Datadog-Agent-What-resources-does-it-consume-
sum:apache.status_2xx_count{role:web} by {env}
EC2
Classic
VPC
Tag-based flexible visualization
Slide 39
Slide 39 text
http://qiita.com/takus/items/c1d71bfcc231d0c24e0a
Slide 40
Slide 40 text
Migrate from EC2 Classic to VPC with Route53
Slide 41
Slide 41 text
role = web
env = classic
role = web
env = vpc
Provision clusters with Chef
Slide 42
Slide 42 text
role = web
env = classic
role = web
env = vpc
Aggregate metrics with 'env' tag
Slide 43
Slide 43 text
Check clusters by Host Maps
https://www.datadoghq.com/blog/2015/03/introducing-host-maps-know-thy-infrastructure/
Slide 44
Slide 44 text
Build your own dashboards
Slide 45
Slide 45 text
PagerDuty Integration
Only pager for related teams (members)
Slide 46
Slide 46 text
What's great?
• No dedicated Ops required for system monitoring
• chef cookbook & built-in integration
• customisable dashboard for each developers
• Find the root cause quickly
• gather all of metrics/events/alerting at one place
• tag-based flexible visualization
• correlating metrics & events
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
Chartio
• Dashboard as a Service
• create beautiful dashboards
• provide connectors to various data sources
• MySQL, Redshift, Google Analytics, BigQuery, etc...
• Presto with Prestogres
• non-engineer can create dashboard w/o SQL
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
How to create charts?
1. Build a query to data source
(drag & drop / SQL)
2. Modify data
(filterɺsortɺadd column)
3. Visualise it
(table, pie chart, bar chart)
Slide 51
Slide 51 text
Connecting to Presto
https://github.com/treasure-data/prestogres
http://docs.treasuredata.com/articles/chartio
Slide 52
Slide 52 text
Join BigQuery & MySQL
Layer1 BigQuery Result
article_id country pv
1 JP 1000000
2 US 2000000
3 GB 3000000
4 JP 4000000
article_id url title
1 http://aaa Article A
2 http://bbb Article B
3 http://ccc Article C
4 http://ddd Article D
Layer2 MySQL Result
Join
https://support.chartio.com/docs/charts/#merging-data-joins
Slide 53
Slide 53 text
Join BigQuery & MySQL
article_id country pv url title
1 JP 1000000 http://aaa Article A
2 US 2000000 http://bbb Article B
3 GB 3000000 http://ccc Article C
4 JP 4000000 http://ddd Article D
Joined Layer : BigQuery + MySQL
https://support.chartio.com/docs/charts/#merging-data-joins
CSV
Graph
Mail
Slide 54
Slide 54 text
Categorical Drop down
Could you give me the ios's article ranking of bbc.co.uk in UK?
e.g. : A request from BizDev team
Slide 55
Slide 55 text
How is the result of a A/B test in California?
It seems that iPhone 6's user complaints about new features.
Can I compare with the result of each devices?
e.g. : A request from client developers
Categorical Drop down
Slide 56
Slide 56 text
What's great?
• No dedicated SQL Monkey required for visualising
• everyone can be SQL Monkey
• non-engineer can do by WebUI
• Easy to drill down from overview to each users
• overall summaries
• each OS, country, region, language, etc...
Slide 57
Slide 57 text
c.f. About Our Data Platform
https://goo.gl/Rc1uuO http://goo.gl/KL1fpL
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
OneLogin
• Identity as a Service
• provides single sign on to various SaaS
• SAML / Auto complete form with browser extension
• non-frustrated 2FA with mobile app
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
Why SaaS?
Use cases
Relationship
Summary
Slide 63
Slide 63 text
Send a Pull Request
Slide 64
Slide 64 text
Send a Pull Request
• Wrote a fluentd - datadog integration
• only aiming for meeting our requirements
• It became an official integration
• https://www.datadoghq.com/2015/02/monitor-
fluentd-datadog/
Slide 65
Slide 65 text
Hold an User Meetup
http://blog.takus.me/2015/02/26/dd-sushi/
Slide 66
Slide 66 text
Gifts from SaaS
Slide 67
Slide 67 text
Discuss products in conferences
Slide 68
Slide 68 text
Discuss products in conferences
• Fortunately, I had some chances to attend them
• Velocity Conference 2013, 2015
• Amazon re:Invent 2014
• Great time to talk with core developers
• learn new services & new features
• explain our feature requests by myself
• ask their product roadmap
Slide 69
Slide 69 text
New Relic's Service Map
Slide 70
Slide 70 text
Anomaly detection by SumoLogic
Slide 71
Slide 71 text
Future of SaaS
• SaaS is becoming hot area in startup
• 2014/12/11: NewRelic raised $115M in IPO
• 2015/01/29: Datadog raised $31M in Funding
• 2015/06/01: SumoLogic raised $80M in Funding
• SaaS will evolve rapidly with a large amount of funds?
• probably in my opinion
Slide 72
Slide 72 text
Why SaaS?
Use cases
Relationship
Summary
Slide 73
Slide 73 text
SaaS makes life easier
• The real world is complex. We have a lot of tasks to do
• Professionals solve specific issues in behalf of us
• No dedicated engineer is needed for it
• We can focus on our own problems
Slide 74
Slide 74 text
We’re hiring!
Site reliability engineer
iOS/Android engineer
Web application engineer
Productivity engineer
ML/NLP engineer
Growth hack engineer
Ad engineer
http://about.smartnews.com/en/careers/
Slide 75
Slide 75 text
One more thing...
Slide 76
Slide 76 text
Join Our Free Lunch!
We love talking about Technologies, Products, Medias, etc...