Tweet
Tweet

Slide 1

Slide 1 text

Accelerating AdTech on AWS in Japan Pragmatic use-cases Dynalyst / fluct / IM-DMP / UNICORN Eiji Shinohara Amazon Web Services Japan, Solutions Architect August 30, 2017 at MARU180

Slide 2

Slide 2 text

안녕하십니까! @werner: Amazon CTO Piljoong Park-san Eiji Shinohara (篠原 英治) § Twitter: @shinodogg § Blog: shinodogg.com AWS Solutions Architect § Market: AdTech & Startup § Area of Depth: Search Korean Cuisine Lover § 三겹살/삼겹살, 불고기, 김치,,,

Slide 3

Slide 3 text

Agenda Japanese AdTech Industry Japanese AdTech Community AdTech on AWS use-cases in Japan vDynalyst http://www.dynalyst.io vfluct https://fluct.jp vIM-DMP https://corp.intimatemerger.com vUNICORN https://uncn.jp

Slide 4

Slide 4 text

Japanese AdTech Industry

Slide 5

Slide 5 text

Japanese AdTech Industry JP 2016 Internet Ads Market Size Research by CCI http://www.cci.co.jp/news/release/2017_04_17/1.html

Slide 6

Slide 6 text

Japanese AdTech Industry JP 2016 Internet Ads Market Size Research by CCI http://www.cci.co.jp/news/release/2017_04_17/1.html $10 Billion Market Smartphone Shift Smartphone Desktop

Slide 7

Slide 7 text

Japanese AdTech Industry https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016

Slide 8

Slide 8 text

Japanese AdTech Industry https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016 Fragmented and Competitive same as other regions

Slide 9

Slide 9 text

Japanese AdTech Community

Slide 10

Slide 10 text

Japanese AdTech Community AdTech Meetup by AWS in 2016 #AWSAdTechJP

Slide 11

Slide 11 text

Japanese AdTech Community AdTech Meetup by AWS in 2016 #AWSAdTechJP “Digital Marketing” Trend DialogOne “LINE” Business Connect “AdNetwork” Admin Tools http://aws.typepad.com/sajp/2016/07/aws-adtech-jp.html Wrap-up Blog Post

Slide 12

Slide 12 text

Japanese AdTech Community Akiba Lab – Over 800 people in Facebook group

Slide 13

Slide 13 text

Japanese AdTech Community Akiba Lab – Over 800 people in Facebook group アドテク⇒AdTech Akiba Lab is a Japanese AdTech community Big year-end party in Dec 2016 Lightning Talks

Slide 14

Slide 14 text

Japanese AdTech Community #CTONight powered by AWS

Slide 15

Slide 15 text

Japanese AdTech Community #CTONight powered by AWS Japanese Top-Notch CTOs gathering

Slide 16

Slide 16 text

Japanese AdTech Community #CTONight powered by AWS v AdTech CTOs at CTO Night J

Slide 17

Slide 17 text

Japanese AdTech Community AdTech CTOs Round Table v Deeper AdTech Talks!

Slide 18

Slide 18 text

Japanese AdTech on AWS Use-Cases

Slide 19

Slide 19 text

AdTech on AWS Use-Cases in Japan Dynalyst http://www.dynalyst.io v Re-Targeting / Re-Engaging v Japan and U.S. fluct https://fluct.jp v SSP: 30 billion impressions in a month v Ajitofm: Podcast @ VOYAGE GROUP in company bar IM-DMP https://corp.intimatemerger.com v Public DMP v Small Engineering Team delivers Big Result UNICORN https://uncn.jp v Full Automated Marketing Platform v International Engineers in Tokyo

Slide 20

Slide 20 text

CyberAgent - Internet Conglomerate https://www.cyberagent.co.jp/en/

Slide 21

Slide 21 text

CyberAgent - Internet Conglomerate http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf

Slide 22

Slide 22 text

CyberAgent - Internet Conglomerate http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf About 500 million USD Ad Business

Slide 23

Slide 23 text

CyberAgent - AdTech Studio https://adtech.cyberagent.io/ https://www.slideshare.net/InsightTechnology/dbts- tokyo-2016-b15-bigdata-actian-matrix-mapr-cyberagent

Slide 24

Slide 24 text

CyberAgent - AdTech Studio https://adtech.cyberagent.io/ https://www.slideshare.net/InsightTechnology/dbts- tokyo-2016-b15-bigdata-actian-matrix-mapr-cyberagent

Slide 25

Slide 25 text

Dynalyst - Dynamic Retargeting for Game Apps Massive Audience Personalized Engagement AWS Summit Tokyo 2014 AWS Summit Tokyo 2015

Slide 26

Slide 26 text

Dynalyst - Log Processing Architecture

Slide 27

Slide 27 text

Japan US ap-northeast-1 us-east-1 Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster S3 Redshift EMR Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster Dynalyst - Log Processing Architecture

Slide 28

Slide 28 text

Japan US ap-northeast-1 us-east-1 Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster S3 Redshift EMR Up to 100 instances Up to 80 shards KCL on ECS Docker Cluster Dynalyst - Log Processing Architecture Petabyte Scale

Slide 29

Slide 29 text

Dynalyst - Cross Region Replication Real-Time Bidding https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf

Slide 30

Slide 30 text

Dynalyst - Cross Region Replication Real-Time Bidding https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf Network Latency is Critical https://www.mopub.com/resources/mopub-demand/mopub-marketplace-overview/network-infrastructure/

Slide 31

Slide 31 text

Dynalyst - Cross Region Replication DynamoDB Streams Amazon RDS Aurora

Slide 32

Slide 32 text

Dynalyst - Real-Time Bidding Train Model: Spark ML / Save Model: Redis Quick Response to Bid Requests! EMR ElastiCache S3 Bid Request Memcached Redis Aurora DynamoDB

Slide 33

Slide 33 text

Dynalyst - Go Global with AWS! Shuhei Kimura v Moving back and forth from Japan to U.S. v Diving deeply into U.S. AdTech eco-system v Planning to use another AWS region in US West

Slide 34

Slide 34 text

VOYAGE GROUP - fluct: SSP

Slide 35

Slide 35 text

fluct - Serverless Architecture in 2016 Serverless for Analyzing contents vBetter Contents/Context matched Ad delivery https://speakerdeck.com/suzuken/how-to-use-aws-lambda-in-document-processing-pipeline

Slide 36

Slide 36 text

fluct – SSP: 30billion impressions in a month Kenta Suzuki A. Advertising transparency v Players are relying on each other v Preventing unethical actions is an entire industry problem! v Letʼs make the Internet better place J Q. What is the trend in AdTech industry?

Slide 37

Slide 37 text

fluct magazine https://magazine.fluct.jp fluct – SSP: 30billion impressions in a month

Slide 38

Slide 38 text

fluct – SSP: 30billion impressions in a month “ads.txt” aims to increase transparency in the AdTech ecosystem How do we introduce ads.txt? fluct magazine https://magazine.fluct.jp

Slide 39

Slide 39 text

fluct - Ad Verification Architecture

Slide 40

Slide 40 text

Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party service Kinesis fluct - Ad Verification Architecture

Slide 41

Slide 41 text

Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party service Kinesis Speed Layer Batch Layer fluct - Ad Verification Architecture

Slide 42

Slide 42 text

Ad tag ALB ECS Lambda Lambda Amazon ES 3rd Party service Kinesis Running “Golang” application on AWS Lambda w/ Apex fluct - Serverless Architecture

Slide 43

Slide 43 text

Apex – Serverless Infrastructure “build, deploy, and manage AWS Lambda functions with ease” http://apex.run/

Slide 44

Slide 44 text

https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex Apex – Serverless Infrastructure Running Golang on AWS Lambda

Slide 45

Slide 45 text

https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex Running Golang on AWS Lambda Apex – Serverless Infrastructure

Slide 46

Slide 46 text

Tech Podcast - VOYAGE GROUP https://www.instagram.com/p/BXWzZ9ngYLZ/ HUGO (https://gohugo.io/) + Hosting on S3 Ajitofm https://ajito.fm/

Slide 47

Slide 47 text

https://ajito.fm/2/ Tech Podcast - VOYAGE GROUP Running Golang on AWS Lambda v Node.js -> Golang Running Golang as a Child Process Utilize STDIN and STDOUT Sounds like “CGI” in Cloud ERA... http://www.kent-web.com/

Slide 48

Slide 48 text

https://ajito.fm/2/ v Node.js -> Golang Running Golang as a Child Process Utilize STDIN and STDOUT Sounds like “CGI” in Cloud ERA... Popular CGI Examples In 90s… Tech Podcast - VOYAGE GROUP Running Golang on AWS Lambda http://www.kent-web.com/

Slide 49

Slide 49 text

AJITO – VOYAGE GROUP Inside Company Bar

Slide 50

Slide 50 text

Intimate Merger - IM-DMP

Slide 51

Slide 51 text

Intimate Merger - IM-DMP Intimate Merger v Founded in 2013 as a Joint Venture FreakOut: The first DSP in Japan Preferred Infrastructure: Cutting Edge Tech v Shareholders in 2017 FreakOut Holdings: Global Marketing Tech group Dentsu: Worldʼs leading Advertising Agency YJCapital: Yahoo! Japan Corporate Venture Capital

Slide 52

Slide 52 text

Intimate Merger - IM-DMP w/ dentsu v Contribute to Public DMP ”dPublic” by dentsu w/ Yahoo! Japan v Connect to Yahoo! Japan DMP https://corp.intimatemerger.com/archives/1855/

Slide 53

Slide 53 text

Intimate Merger - IM-DMP w/ dentsu v Contribute to Public DMP ”dPublic” by dentsu w/ Yahoo! Japan v Connect to Yahoo! Japan DMP https://corp.intimatemerger.com/archives/1855/ Psychographic Demographic 400 million Audience Data

Slide 54

Slide 54 text

IM-DMP - Architecture

Slide 55

Slide 55 text

IM-DMP - Architecture ECS RDS ElastiCache Spot Fleet ECS CSV TSV JSON SQS IM-DMP UI&API Data Processing S3 3rd Party Partners

Slide 56

Slide 56 text

ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data Processing S3 3rd Party Data Providers Everything is started from S3 upload Simple File-based Trigger Spot Fleet ECS IM-DMP - S3 File-based Architecture

Slide 57

Slide 57 text

ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data Processing S3 3rd Party Partners ECS & Spot Fleet Spot Fleet ECS IM-DMP - Amazon ECS & EC2 Spot Fleet

Slide 58

Slide 58 text

AWS Summit Tokyo 2017 https://www.youtube.com/watch?v=vJZJY_t8JDE https://www.slideshare.net/im_docs/amazon-ecs-spotfleet IM-DMP - Amazon ECS & EC2 Spot Fleet

Slide 59

Slide 59 text

AWS Summit Tokyo 2017 https://www.youtube.com/watch?v=vJZJY_t8JDE https://www.slideshare.net/im_docs/amazon-ecs-spotfleet IM-DMP - Amazon ECS & EC2 Spot Fleet

Slide 60

Slide 60 text

Amazon EC2 Spot Instances July 26, 2017 / ap-northeast-1 / Linux On Demand Reserved Instances for 1 year Spot Instances Spot Block All Upfront Partial Upfront No Upfront 1h 6h c4.large $0.126 $0.084 (33%) $0.086 (32%) $0.090 (29%) $0.029 (77%) $0.077 (39%) $0.098 (22%) m4.large $0.129 $0.081 (37%) $0.083 (36%) $0.087 (32%) $0.027 (79%) $0.101 (21%) $0.128 (0.7%) r3.large $0.20 $0.127 (36%) $0.130 (35%) $0.149 (26%) $0.031 (84%) $0.116 (42%) $0.147 (26%)

Slide 61

Slide 61 text

On Demand Reserved Instances for 1 year Spot Instances Spot Block All Upfront Partial Upfront No Upfront 1h 6h c4.large $0.126 $0.084 (33%) $0.086 (32%) $0.090 (29%) $0.029 (77%) $0.077 (39%) $0.098 (22%) m4.large $0.129 $0.081 (37%) $0.083 (36%) $0.087 (32%) $0.027 (79%) $0.101 (21%) $0.128 (0.7%) r3.large $0.20 $0.127 (36%) $0.130 (35%) $0.149 (26%) $0.031 (84%) $0.116 (42%) $0.147 (26%) July 26, 2017 / ap-northeast-1 / Linux Amazon EC2 Spot Instances

Slide 62

Slide 62 text

ECS RDS ElastiCache CSV TSV JSON SQS IM-DMP UI&API Data Processing S3 3rd Party Partners Spot Instances Spot Fleet ECS IM-DMP - Elasticsearch on Spot Instances

Slide 63

Slide 63 text

Elasticsearch: Approx. 400 million IDs v Extract IDs with v Keyword (by browsing history) v Segment v User Agent v IP address v Geo https://www.slideshare.net/im_docs/elasticsearch-48873206 IM-DMP - Elasticsearch on Spot Instances

Slide 64

Slide 64 text

Elasticsearch on Spot Instances v approx. 500vCPUs for Analytics workload Over 8vCPUs i3 Instances IM-DMP - Elasticsearch on Spot Instances

Slide 65

Slide 65 text

Intimate Merger Small engineering team delivers Big results! Masafumi Osogoe & Kazuki Matsuda Kazuki Masafumi

Slide 66

Slide 66 text

Bulbit - Adways company http://adways.kr/ https://www.adways.net

Slide 67

Slide 67 text

Bulbit - Party Track & UNICORN

Slide 68

Slide 68 text

68 UNICORN - Full Automated Marketing Platform

Slide 69

Slide 69 text

Greatly Skilled Engineers from China J vHailin Hu vXiaoyi Qu UNICORN - Full Automated Marketing Platform ü How do you feel about working on AdTech in Japan? ü What are you focusing on? Hailin Xiaoyi

Slide 70

Slide 70 text

Greatly Skilled Engineers from China J vHailin Hu vXiaoyi Qu UNICORN - Full Automated Marketing Platform Itʼs like a “Gold Mine” ü Day-by-Day Evolution ü Achieving Goals with latest Big Data Technologies ü Utilize “Amazon Athena” in a massive way! Hailin Xiaoyi

Slide 71

Slide 71 text

UNICORN - Architecture

Slide 72

Slide 72 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Architecture

Slide 73

Slide 73 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Real-Time Bidding From Ruby to Golang “Speed is King” in Real-Time Bidding

Slide 74

Slide 74 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Data Analysis v Extract data for Machine Learning every 30min v Ad-Hoc Big Data Analysis

Slide 75

Slide 75 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Machine Learning v w/ Minimum Libraries ü No Heavy Framework ü As Fast As Possible!! v Making Steady Effort ü Plan-Do-Check-Act ü Parameter Tuning ü A/B Testing

Slide 76

Slide 76 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Machine Learning For Real-Time Bidding, Bidding servers load “Trained Models” into Memory

Slide 77

Slide 77 text

Auto Scaling Up to 200 instances Athena Redshift Deep Learning on EC2 S3 UNICORN - Big Data Technology v Right Technology in the Right Place v Recently in favor with “Apache Flink”

Slide 78

Slide 78 text

AdTech use-cases in Japan

Slide 79

Slide 79 text

Let's meet up at re:Invent 2017!! https://reinvent.awsevents.com/

Slide 80

Slide 80 text

Let's meet up at re:Invent 2017!! https://reinvent.awsevents.com/

Slide 81

Slide 81 text

Let's meet up at re:Invent 2017!! https://reinvent.awsevents.com/ @Kimchi Korean BBQ http://www.kimchilasvegas.com/

Slide 82

Slide 82 text

감사합니다!!