Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why and How SmartNews uses SaaS?

Why and How SmartNews uses SaaS?

Takumi Sakamoto

June 26, 2015

More Decks by Takumi Sakamoto

Other Decks in Programming


  1. Why and How SmartNews uses SaaS? @takus 2015.06.26 SmartNews Official

    Character Chikyu-Kun
  2. Takumi Sakamoto (@takus)

  3. None
  4. SmartNews Crosses 10 Million Downloads in Global Source: App Annie

  5. SmartNews has No.1 MAU in News Apps, Japan Comparison of

    the other News Apps 4.3M MAU Source: Nielsen Japan
  6. Ranking of Active Users  * 2 5 1 1 199

        -‐‑‒ E . 1 2 3 4 5 6 7 8 9 10 1 B 1 2 A 0 1 5
  7. Long Time Spent Facebook messenger LINE Facebook Gmail Youtube COOKPAD

    News App A SmartNews News App B News App C ※ Spending minutes/person, month Source: Nielsen Japan
  8. Why SaaS? Use cases Relationship Summary

  9. We want to focus on OUR PRODUCT

  10. None
  11. We must move fast

  12. None
  13. Engineering resources are limited

  14. Efficient Ads System Document Classification Site Reliability Engineering High Performance

    API Server News Crawler Realtime Ranking Algorithm Data Management Platform UI/UX
  15. Can we assign excellent engineers to every positions?

  16. No, probably

  17. The answer is

  18. None
  19. Why SaaS? • Solves specific issues for us • monitoring,

    alerting, visualising etc... • Less bootstrap process • tiny setup process (a java agent, a chef recipe etc...) • built-in user friendly web Interface • No dedicated person is needed to develop/operate • experts in an area do it well • easy to manage with programable API
  20. Why SaaS? Use cases Relationship Summary

  21. None
  22. NewRelic • (Application) Monitoring as a Service • easy setup

    for popular language (e.g. Java) • add method traces by annotation • trace SQL queries automatically • records deployments
  23. Setup for Tomcat • deploy newrelic archives to $CATALINA_HOME •

    edit newrelic.yml • application name • add -javaagent • $CATALINA_HOME/newrelic/newrelic.jar • add environment if necessary • -Dnewrelic.environment=xxx
  24. Customize for each Apps # add method tracer @Trace(metricName="API/yourMethod", dispatcher=true)

    public Response yourMethod() { ... ... ... } # add custom metric MetricAggregator aggregator = NewRelic.getAgent().getMetricAggregator(); aggregator.recordMetric(key, value); # record exception NewRelic.noticeError(ex);
  25. Trace methods & queries

  26. Record Deployments

  27. from fabric.api run from fabric.decorators import runs_once, parallel @parallel @task

    def deploy(): do_your_task() notify() @runs_once def notify(): ver = release_version() run("java -jar newrelic.jar deployment --appname=xxxx --revision=%s" % ver) Record deployment by fabric https://docs.newrelic.com/docs/agents/java-agent/instrumentation/recording-deployments-java-agent
  28. NewRelic Insights Query to custom metrics via NRQL SmartNews seems

    top 10 user of Insights
  29. What's great? • Developer can monitor their own app by

    themselves • adding custom tracer, custom metrics • Don't have to prepare profiler for each language • Java, Scala, Ruby, (Golang), etc... • suit for Microservices
  30. None
  31. Datadog • (System) Monitoring as a Service • easy to

    setup with Chef • 100+ built in integration (AWS, MySQL, Docker...) • correlate metrics with events • tag-based flexible visualization • build your own interactive dashboards
  32. Setup dd-agent with Chef # Install cookbook knife cookbook site

    install datadog # Set Datadog-specific attributes node.default['datadog']['api_key'] = "xxxxxxxxxx" # Add a recipe in your run list "run_list": ["recipe[datadog::dd-agent]"]
  33. BTW, What is dd-agent? http://help.datadoghq.com/hc/en-us/articles/203034929-What-is-the-Datadog-Agent-What-resources-does-it-consume-

  34. Built-in Integrations

  35. Setup MySQL Monitor # Edit conf.d/mysql.yaml instances: - server: localhost

    user: datadog pass: xxxxxxxx tags: - dbtype:master options: replication: 0
  36. Correlate metrics w/ deployment

  37. New Relic Integration

  38. sum:apache.status_2xx_count{role:web} by {env} EC2 Classic VPC Tag-based flexible visualization

  39. http://qiita.com/takus/items/c1d71bfcc231d0c24e0a

  40. Migrate from EC2 Classic to VPC with Route53

  41. role = web env = classic role = web env

    = vpc Provision clusters with Chef
  42. role = web env = classic role = web env

    = vpc Aggregate metrics with 'env' tag
  43. Check clusters by Host Maps https://www.datadoghq.com/blog/2015/03/introducing-host-maps-know-thy-infrastructure/

  44. Build your own dashboards

  45. PagerDuty Integration Only pager for related teams (members)

  46. What's great? • No dedicated Ops required for system monitoring

    • chef cookbook & built-in integration • customisable dashboard for each developers • Find the root cause quickly • gather all of metrics/events/alerting at one place • tag-based flexible visualization • correlating metrics & events
  47. None
  48. Chartio • Dashboard as a Service • create beautiful dashboards

    • provide connectors to various data sources • MySQL, Redshift, Google Analytics, BigQuery, etc... • Presto with Prestogres • non-engineer can create dashboard w/o SQL
  49. None
  50. How to create charts? 1. Build a query to data

    source (drag & drop / SQL) 2. Modify data (filterɺsortɺadd column) 3. Visualise it (table, pie chart, bar chart)
  51. Connecting to Presto https://github.com/treasure-data/prestogres http://docs.treasuredata.com/articles/chartio

  52. Join BigQuery & MySQL Layer1 BigQuery Result article_id country pv

    1 JP 1000000 2 US 2000000 3 GB 3000000 4 JP 4000000 article_id url title 1 http://aaa Article A 2 http://bbb Article B 3 http://ccc Article C 4 http://ddd Article D Layer2 MySQL Result Join https://support.chartio.com/docs/charts/#merging-data-joins
  53. Join BigQuery & MySQL article_id country pv url title 1

    JP 1000000 http://aaa Article A 2 US 2000000 http://bbb Article B 3 GB 3000000 http://ccc Article C 4 JP 4000000 http://ddd Article D Joined Layer : BigQuery + MySQL https://support.chartio.com/docs/charts/#merging-data-joins CSV Graph Mail
  54. Categorical Drop down Could you give me the ios's article

    ranking of bbc.co.uk in UK? e.g. : A request from BizDev team
  55. How is the result of a A/B test in California?

    It seems that iPhone 6's user complaints about new features. Can I compare with the result of each devices? e.g. : A request from client developers Categorical Drop down
  56. What's great? • No dedicated SQL Monkey required for visualising

    • everyone can be SQL Monkey • non-engineer can do by WebUI • Easy to drill down from overview to each users • overall summaries • each OS, country, region, language, etc...
  57. c.f. About Our Data Platform https://goo.gl/Rc1uuO http://goo.gl/KL1fpL

  58. None
  59. OneLogin • Identity as a Service • provides single sign

    on to various SaaS • SAML / Auto complete form with browser extension • non-frustrated 2FA with mobile app
  60. None
  61. None
  62. Why SaaS? Use cases Relationship Summary

  63. Send a Pull Request

  64. Send a Pull Request • Wrote a fluentd - datadog

    integration • only aiming for meeting our requirements • It became an official integration • https://www.datadoghq.com/2015/02/monitor- fluentd-datadog/
  65. Hold an User Meetup http://blog.takus.me/2015/02/26/dd-sushi/

  66. Gifts from SaaS

  67. Discuss products in conferences

  68. Discuss products in conferences • Fortunately, I had some chances

    to attend them • Velocity Conference 2013, 2015 • Amazon re:Invent 2014 • Great time to talk with core developers • learn new services & new features • explain our feature requests by myself • ask their product roadmap
  69. New Relic's Service Map

  70. Anomaly detection by SumoLogic

  71. Future of SaaS • SaaS is becoming hot area in

    startup • 2014/12/11: NewRelic raised $115M in IPO • 2015/01/29: Datadog raised $31M in Funding • 2015/06/01: SumoLogic raised $80M in Funding • SaaS will evolve rapidly with a large amount of funds? • probably in my opinion
  72. Why SaaS? Use cases Relationship Summary

  73. SaaS makes life easier • The real world is complex.

    We have a lot of tasks to do • Professionals solve specific issues in behalf of us • No dedicated engineer is needed for it • We can focus on our own problems
  74. We’re hiring! Site reliability engineer iOS/Android engineer Web application engineer

    Productivity engineer ML/NLP engineer Growth hack engineer Ad engineer http://about.smartnews.com/en/careers/
  75. One more thing...

  76. Join Our Free Lunch! We love talking about Technologies, Products,

    Medias, etc...