Accelerating an Ad-Tech Service with OpenStack Cloud CyberAgent, Inc Makoto Hasegawa Ryo Tagami @OpenStack Summit 2015 Vancouver

We came from Tokyo !! Hello Vancouver !! Tokyo Vancouver

Agenda About CyberAgent OpenStack in CyberAgent What is required for infrastructure of Ad-Tech What is Ad-Tech in CyberAgent Why we choose OpenStack Big picture of our private cloud Deployment / Operation / Monitoring Future of our private cloud

Who are you ?

About us Makoto Hasegawa I am a cloud architect and leading the system admin team of CyberAgent Ad- Tech Business Division. I have been managing the OpenStack for about one year. Other than OpenStack, I also use cloud platforms such as aws and gcp. Recently I’m most interested in how I can automate the cloud management by using various different tools. and … @makocchi https://www.facebook.com/makocchi0923

About us Ryo Tagami I’m an Infrastructure Engineer for Ad-Tech Business Division at CyberAgent Inc., where I design and deploy OpenStack based private cloud which hosts multiple advertisement related services CyberAgent Inc. offers. Keeping it neat.

CyberAgent ?

“To create the 21st century's leading company” CyberAgent is expanding its business in the field of Internet, a leading industry of the 21st Century.

About CyberAgent We provide a variety of services such as on-line games, on-line communities and Ad-Tech. And we have been running our Ad-Tech services on our OpenStack Cloud. and m ore !!

What is Ad-Tech in CyberAgent

The Basic Structure of Ad-Tech https://www.cyberagent.co.jp/en/ir/personal/adtech/adtech_05/ Ad-Tech is composed of many different platforms such as DSP, DMP, SSP, depending on each other.

CyberAgent’s Ad-Tech Business https://www.cyberagent.co.jp/en/ir/personal/adtech/adtech_05/ Each component is provided by different players, separate companies like this. This diagram does not show all the relative companies and players and there are many more. Also new platforms are added day by day.

What is required for infrastructure of Ad-Tech

The infrastructure of Ad-Tech has to be … Flexible, Agile and Stable more Flexible ! There are many platforms for services exists, and the infrastructure under them have to work with all of them. more Agile ! The infrastructure has to be prepared quickly and deleted when they are not necessary any more. more Stable ! One small trouble and system down has huge impact. So we need flexible, agile and stable Cloud platform.

Why we choose OpenStack

There are 3 reasons why 1 2 3 OpenStack is opensource. CyberAgent has strong culture of leveraging opensource technologies. Using OSS benefits us in terms of catching up new technologies and also cutting costs. There was a strong momentum that OpenStack should be the next mainstream of Cloud Management System when we are evaluating several options. I thought the technical skills and the motivation of our engineers will improve by learning, deploying and operating OpenStack, which contains a lot of different technologies

OpenStack in CyberAgent

2012.09 Folsom 2013.04 Grizzly 2013.10 Havana 2014.04 Icehouse 2014.10 Juno 2015.04 Kilo We used OpenStack only for PoC We provided over 10 services on OpenStack Grizzly (10+ compute nodes / 10 engineers) 2013.10 Ad-Tech Division started in CyberAgent 2014.06 We provided some Ad-Tech services on OpenStack Icehouse (40+ compute nodes / 3 engineers) 2015.03 We provided some Ad-Tech services on OpenStack Juno (100+ compute nodes / 3 engineers) We will start testing so very soon

Big picture of our private cloud

Big picture of our private cloud Upgrade or new deployment? Multiple OpenStack deployments. Why? Difficult to upgrade (safely). Deploy a new cluster, abandon the old one.

Big picture of our private cloud Codename: GAIA Codename: nevera Icehouse Production Codename: galadeira Icehouse Production (Sandbox) Codename: diana Juno Production Codename: minerva Icehouse Personal Development Codename: venus Juno Personal Development (Future) Codename: vesta Kilo Production (Future) Codename: eiskeller Icehouse Personal Development (Sandbox)

Big picture of our private cloud Number of Computes 200+ computes Number of CPU cores 5,000 cores / 10,000 threads Number of VM Instances 1,000+ instances Network Dual 10G from server to ToR Dual 40G from ToR to EoR Specs of diana

Big picture of our private cloud OpenStack components: • Keystone • Glance • Nova • Neutron • Swift Proxy • Cinder • Heat • Ceilometer Ceph is backing: • Glance • Swift Proxy • Cinder Specs of diana

Big picture of our private cloud • Redundancy • Hardware • RAID5 SSDs • Redundant power supply • Redundant network connection • Software • Load-balanced API endpoints • MariaDB Galera Cluster • RabbitMQ HA Cluster

Deployment ɹOperation ɹɹMonitoring

Deployment / Operation / Monitoring Tools? Ansible In-house playbook and roles May release them on github.com

Deployment / Operation / Monitoring Sending all logs to central log server With rsyslog For troubleshooting using grep

Deployment / Operation / Monitoring Basic Service Monitoring • Process Existence • TCP Port State API Monitoring • Response Time • High Level Function Testing Standard Monitoring • Hardware Health • OS Resources Templates and scripts may be released

Compromises In-house tool had to be developed Load balancer management Reason: Not using Neutron LBaaS at the moment Network design differs from reference implementation LBaaS driver still in development for Juno Solution: Manage load balancer outside Neutron (with some level of multi tenancy) Instance tagging & filtering Reason: High demand from users migrating from AWS Not implemented in Juno Solution: Use instance ‘metadata’ as tag field

Future of our private cloud

Bucket List Software Defined Network Get rid of conventional VLAN based networking,
 introduce IP-based overlay networking Experimenting “MidoNet” by Midokura.

Bucket List Software Defined Storage Decouple storage from compute End-to-End 40G Networking SDS, in memory KVS Ironic (Baremetal) Virtual machines does not cover every use case Kilo and Liberty New is always better.

Any Questions ?

Thank you very match !! Today’s presentation will be available on our Github. https://github.com/CyberAgent/openstack-summit-2015-vancouver/ If you have any questions and impressions, please create issue on github !! we reply as soon as possible. https://goo.gl/6dsfSk See you next Tokyo summit !!