Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Accelarating an Ad-Tech service with OpenStack cloud

Accelarating an Ad-Tech service with OpenStack cloud

Accelarating an Ad-Tech service with OpenStack cloud


May 20, 2015

More Decks by makocchi

Other Decks in Technology


  1. Accelerating an Ad-Tech Service with OpenStack Cloud CyberAgent, Inc Makoto

    Hasegawa Ryo Tagami @OpenStack Summit 2015 Vancouver
  2. Agenda About CyberAgent OpenStack in CyberAgent What is required for

    infrastructure of Ad-Tech What is Ad-Tech in CyberAgent Why we choose OpenStack Big picture of our private cloud Deployment / Operation / Monitoring Future of our private cloud
  3. About us Makoto Hasegawa I am a cloud architect and

    leading the system admin team of CyberAgent Ad- Tech Business Division. I have been managing the OpenStack for about one year. Other than OpenStack, I also use cloud platforms such as aws and gcp. Recently I’m most interested in how I can automate the cloud management by using various different tools. and … @makocchi https://www.facebook.com/makocchi0923
  4. About us Ryo Tagami I’m an Infrastructure Engineer for Ad-Tech

    Business Division at CyberAgent Inc., where I design and deploy OpenStack based private cloud which hosts multiple advertisement related services CyberAgent Inc. offers. Keeping it neat.
  5. “To create the 21st century's leading company” CyberAgent is expanding

    its business in the field of Internet, a leading industry of the 21st Century.
  6. About CyberAgent We provide a variety of services such as

    on-line games, on-line communities and Ad-Tech. And we have been running our Ad-Tech services on our OpenStack Cloud. and m ore !!
  7. CyberAgent’s Ad-Tech Business https://www.cyberagent.co.jp/en/ir/personal/adtech/adtech_05/ Each component is provided by different

    players, separate companies like this. This diagram does not show all the relative companies and players and there are many more. Also new platforms are added day by day.
  8. The infrastructure of Ad-Tech has to be … Flexible, Agile

    and Stable more Flexible ! There are many platforms for services exists, and the infrastructure under them have to work with all of them. more Agile ! The infrastructure has to be prepared quickly and deleted when they are not necessary any more. more Stable ! One small trouble and system down has huge impact. So we need flexible, agile and stable Cloud platform.
  9. There are 3 reasons why 1 2 3 OpenStack is

    opensource. CyberAgent has strong culture of leveraging opensource technologies. Using OSS benefits us in terms of catching up new technologies and also cutting costs. There was a strong momentum that OpenStack should be the next mainstream of Cloud Management System when we are evaluating several options. I thought the technical skills and the motivation of our engineers will improve by learning, deploying and operating OpenStack, which contains a lot of different technologies
  10. 2012.09 Folsom 2013.04 Grizzly 2013.10 Havana 2014.04 Icehouse 2014.10 Juno

    2015.04 Kilo We used OpenStack only for PoC We provided over 10 services on OpenStack Grizzly (10+ compute nodes / 10 engineers) 2013.10 Ad-Tech Division started in CyberAgent 2014.06 We provided some Ad-Tech services on OpenStack Icehouse (40+ compute nodes / 3 engineers) 2015.03 We provided some Ad-Tech services on OpenStack Juno (100+ compute nodes / 3 engineers) We will start testing so very soon
  11. Big picture of our private cloud Upgrade or new deployment?

    Multiple OpenStack deployments. Why? Difficult to upgrade (safely). Deploy a new cluster, abandon the old one.
  12. Big picture of our private cloud Codename: GAIA Codename: nevera

    Icehouse Production Codename: galadeira Icehouse Production (Sandbox) Codename: diana Juno Production Codename: minerva Icehouse Personal Development Codename: venus Juno Personal Development (Future) Codename: vesta Kilo Production (Future) Codename: eiskeller Icehouse Personal Development (Sandbox)
  13. Big picture of our private cloud Number of Computes 200+

    computes Number of CPU cores 5,000 cores / 10,000 threads Number of VM Instances 1,000+ instances Network Dual 10G from server to ToR Dual 40G from ToR to EoR Specs of diana
  14. Big picture of our private cloud OpenStack components: • Keystone

    • Glance • Nova • Neutron • Swift Proxy • Cinder • Heat • Ceilometer Ceph is backing: • Glance • Swift Proxy • Cinder Specs of diana
  15. Big picture of our private cloud • Redundancy • Hardware

    • RAID5 SSDs • Redundant power supply • Redundant network connection • Software • Load-balanced API endpoints • MariaDB Galera Cluster • RabbitMQ HA Cluster
  16. Deployment / Operation / Monitoring Sending all logs to central

    log server With rsyslog For troubleshooting using grep
  17. Deployment / Operation / Monitoring Basic Service Monitoring • Process

    Existence • TCP Port State API Monitoring • Response Time • High Level Function Testing Standard Monitoring • Hardware Health • OS Resources Templates and scripts may be released
  18. Compromises In-house tool had to be developed Load balancer management

    Reason: Not using Neutron LBaaS at the moment Network design differs from reference implementation LBaaS driver still in development for Juno Solution: Manage load balancer outside Neutron (with some level of multi tenancy) Instance tagging & filtering Reason: High demand from users migrating from AWS Not implemented in Juno Solution: Use instance ‘metadata’ as tag field
  19. Bucket List Software Defined Network Get rid of conventional VLAN

    based networking,
 introduce IP-based overlay networking Experimenting “MidoNet” by Midokura.
  20. Bucket List Software Defined Storage Decouple storage from compute End-to-End

    40G Networking SDS, in memory KVS Ironic (Baremetal) Virtual machines does not cover every use case Kilo and Liberty New is always better.
  21. Thank you very match !! Today’s presentation will be available

    on our Github. https://github.com/CyberAgent/openstack-summit-2015-vancouver/ If you have any questions and impressions, please create issue on github !! we reply as soon as possible. https://goo.gl/6dsfSk See you next Tokyo summit !!