Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Accelerating AdTech on AWS in Japan

Eiji Shinohara
September 13, 2017

Accelerating AdTech on AWS in Japan

Japanese AdTech Industry, Community, and Use-cases on AWS -
Dynalyst, fluct, IM-DMP, UNICORN.
At "AdTech on AWS" event in Seoul on August 30th in 2017 https://aws.amazon.com/ko/events/seminars/ad-tech-on-aws-011/
#AWSAdTechJP

Eiji Shinohara

September 13, 2017
Tweet

More Decks by Eiji Shinohara

Other Decks in Technology

Transcript

  1. Accelerating AdTech on AWS in Japan Pragmatic use-cases Dynalyst /

    fluct / IM-DMP / UNICORN Eiji Shinohara Amazon Web Services Japan, Solutions Architect August 30, 2017 at MARU180
  2. 안녕하십니까! @werner: Amazon CTO Piljoong Park-san Eiji Shinohara (篠原 英治)

    § Twitter: @shinodogg § Blog: shinodogg.com AWS Solutions Architect § Market: AdTech & Startup § Area of Depth: Search Korean Cuisine Lover § 三겹살/삼겹살, 불고기, 김치,,,
  3. Agenda Japanese AdTech Industry Japanese AdTech Community AdTech on AWS

    use-cases in Japan vDynalyst http://www.dynalyst.io vfluct https://fluct.jp vIM-DMP https://corp.intimatemerger.com vUNICORN https://uncn.jp
  4. Japanese AdTech Industry JP 2016 Internet Ads Market Size Research

    by CCI http://www.cci.co.jp/news/release/2017_04_17/1.html
  5. Japanese AdTech Industry JP 2016 Internet Ads Market Size Research

    by CCI http://www.cci.co.jp/news/release/2017_04_17/1.html $10 Billion Market Smartphone Shift Smartphone Desktop
  6. Japanese AdTech Community AdTech Meetup by AWS in 2016 #AWSAdTechJP

    “Digital Marketing” Trend DialogOne “LINE” Business Connect “AdNetwork” Admin Tools http://aws.typepad.com/sajp/2016/07/aws-adtech-jp.html Wrap-up Blog Post
  7. Japanese AdTech Community Akiba Lab – Over 800 people in

    Facebook group アドテク⇒AdTech Akiba Lab is a Japanese AdTech community Big year-end party in Dec 2016 Lightning Talks
  8. AdTech on AWS Use-Cases in Japan Dynalyst http://www.dynalyst.io v Re-Targeting

    / Re-Engaging v Japan and U.S. fluct https://fluct.jp v SSP: 30 billion impressions in a month v Ajitofm: Podcast @ VOYAGE GROUP in company bar IM-DMP https://corp.intimatemerger.com v Public DMP v Small Engineering Team delivers Big Result UNICORN https://uncn.jp v Full Automated Marketing Platform v International Engineers in Tokyo
  9. Dynalyst - Dynamic Retargeting for Game Apps Massive Audience Personalized

    Engagement AWS Summit Tokyo 2014 AWS Summit Tokyo 2015
  10. Japan US ap-northeast-1 us-east-1 Up to 100 instances Up to

    80 shards KCL on ECS Docker Cluster S3 Redshift EMR Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster Dynalyst - Log Processing Architecture
  11. Japan US ap-northeast-1 us-east-1 Up to 100 instances Up to

    80 shards KCL on ECS Docker Cluster S3 Redshift EMR Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster Dynalyst - Log Processing Architecture Petabyte Scale
  12. Dynalyst - Real-Time Bidding Train Model: Spark ML / Save

    Model: Redis Quick Response to Bid Requests! EMR ElastiCache S3 Bid Request Memcached Redis Aurora DynamoDB
  13. Dynalyst - Go Global with AWS! Shuhei Kimura v Moving

    back and forth from Japan to U.S. v Diving deeply into U.S. AdTech eco-system v Planning to use another AWS region in US West
  14. fluct - Serverless Architecture in 2016 Serverless for Analyzing contents

    vBetter Contents/Context matched Ad delivery https://speakerdeck.com/suzuken/how-to-use-aws-lambda-in-document-processing-pipeline
  15. fluct – SSP: 30billion impressions in a month Kenta Suzuki

    A. Advertising transparency v Players are relying on each other v Preventing unethical actions is an entire industry problem! v Letʼs make the Internet better place J Q. What is the trend in AdTech industry?
  16. fluct – SSP: 30billion impressions in a month “ads.txt” aims

    to increase transparency in the AdTech ecosystem How do we introduce ads.txt? fluct magazine https://magazine.fluct.jp
  17. Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party

    service Kinesis fluct - Ad Verification Architecture
  18. Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party

    service Kinesis Speed Layer Batch Layer fluct - Ad Verification Architecture
  19. Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party

    service Kinesis Running “Golang” application on AWS Lambda w/ Apex fluct - Serverless Architecture
  20. https://ajito.fm/2/ Tech Podcast - VOYAGE GROUP Running Golang on AWS

    Lambda v Node.js -> Golang Running Golang as a Child Process Utilize STDIN and STDOUT Sounds like “CGI” in Cloud ERA... http://www.kent-web.com/
  21. https://ajito.fm/2/ v Node.js -> Golang Running Golang as a Child

    Process Utilize STDIN and STDOUT Sounds like “CGI” in Cloud ERA... Popular CGI Examples In 90s… Tech Podcast - VOYAGE GROUP Running Golang on AWS Lambda http://www.kent-web.com/
  22. Intimate Merger - IM-DMP Intimate Merger v Founded in 2013

    as a Joint Venture FreakOut: The first DSP in Japan Preferred Infrastructure: Cutting Edge Tech v Shareholders in 2017 FreakOut Holdings: Global Marketing Tech group Dentsu: Worldʼs leading Advertising Agency YJCapital: Yahoo! Japan Corporate Venture Capital
  23. Intimate Merger - IM-DMP w/ dentsu v Contribute to Public

    DMP ”dPublic” by dentsu w/ Yahoo! Japan v Connect to Yahoo! Japan DMP https://corp.intimatemerger.com/archives/1855/
  24. Intimate Merger - IM-DMP w/ dentsu v Contribute to Public

    DMP ”dPublic” by dentsu w/ Yahoo! Japan v Connect to Yahoo! Japan DMP https://corp.intimatemerger.com/archives/1855/ Psychographic Demographic 400 million Audience Data
  25. IM-DMP - Architecture ECS RDS ElastiCache Spot Fleet ECS CSV

    TSV JSON SQS IM-DMP UI&API Data Processing S3 3rd Party Partners
  26. ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data

    Processing S3 3rd Party Data Providers Everything is started from S3 upload Simple File-based Trigger Spot Fleet ECS IM-DMP - S3 File-based Architecture
  27. ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data

    Processing S3 3rd Party Partners ECS & Spot Fleet Spot Fleet ECS IM-DMP - Amazon ECS & EC2 Spot Fleet
  28. Amazon EC2 Spot Instances July 26, 2017 / ap-northeast-1 /

    Linux On Demand Reserved Instances for 1 year Spot Instances Spot Block All Upfront Partial Upfront No Upfront 1h 6h c4.large $0.126 $0.084 (33%) $0.086 (32%) $0.090 (29%) $0.029 (77%) $0.077 (39%) $0.098 (22%) m4.large $0.129 $0.081 (37%) $0.083 (36%) $0.087 (32%) $0.027 (79%) $0.101 (21%) $0.128 (0.7%) r3.large $0.20 $0.127 (36%) $0.130 (35%) $0.149 (26%) $0.031 (84%) $0.116 (42%) $0.147 (26%)
  29. On Demand Reserved Instances for 1 year Spot Instances Spot

    Block All Upfront Partial Upfront No Upfront 1h 6h c4.large $0.126 $0.084 (33%) $0.086 (32%) $0.090 (29%) $0.029 (77%) $0.077 (39%) $0.098 (22%) m4.large $0.129 $0.081 (37%) $0.083 (36%) $0.087 (32%) $0.027 (79%) $0.101 (21%) $0.128 (0.7%) r3.large $0.20 $0.127 (36%) $0.130 (35%) $0.149 (26%) $0.031 (84%) $0.116 (42%) $0.147 (26%) July 26, 2017 / ap-northeast-1 / Linux Amazon EC2 Spot Instances
  30. ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data

    Processing S3 3rd Party Partners Spot Instances Spot Fleet ECS IM-DMP - Elasticsearch on Spot Instances
  31. Elasticsearch: Approx. 400 million IDs v Extract IDs with v

    Keyword (by browsing history) v Segment v User Agent v IP address v Geo https://www.slideshare.net/im_docs/elasticsearch-48873206 IM-DMP - Elasticsearch on Spot Instances
  32. Elasticsearch on Spot Instances v approx. 500vCPUs for Analytics workload

    Over 8vCPUs i3 Instances IM-DMP - Elasticsearch on Spot Instances
  33. Greatly Skilled Engineers from China J vHailin Hu vXiaoyi Qu

    UNICORN - Full Automated Marketing Platform ü How do you feel about working on AdTech in Japan? ü What are you focusing on? Hailin Xiaoyi
  34. Greatly Skilled Engineers from China J vHailin Hu vXiaoyi Qu

    UNICORN - Full Automated Marketing Platform Itʼs like a “Gold Mine” ü Day-by-Day Evolution ü Achieving Goals with latest Big Data Technologies ü Utilize “Amazon Athena” in a massive way! Hailin Xiaoyi
  35. Auto Scaling Up to 200 instances Athena Redshift Deep Learning

    on EC2 S3 UNICORN - Real-Time Bidding From Ruby to Golang “Speed is King” in Real-Time Bidding
  36. Auto Scaling Up to 200 instances Athena Redshift Deep Learning

    on EC2 S3 UNICORN - Data Analysis v Extract data for Machine Learning every 30min v Ad-Hoc Big Data Analysis
  37. Auto Scaling Up to 200 instances Athena Redshift Deep Learning

    on EC2 S3 UNICORN - Machine Learning v w/ Minimum Libraries ü No Heavy Framework ü As Fast As Possible!! v Making Steady Effort ü Plan-Do-Check-Act ü Parameter Tuning ü A/B Testing
  38. Auto Scaling Up to 200 instances Athena Redshift Deep Learning

    on EC2 S3 UNICORN - Machine Learning For Real-Time Bidding, Bidding servers load “Trained Models” into Memory
  39. Auto Scaling Up to 200 instances Athena Redshift Deep Learning

    on EC2 S3 UNICORN - Big Data Technology v Right Technology in the Right Place v Recently in favor with “Apache Flink”