Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Multi-location Implementation of OpenStack

Naoto Gohko
October 27, 2015

Multi-location Implementation of OpenStack

http://sched.co/49t1

ConoHa supports multi locations which are Japan(Tokyo), USA(San Jose) and Singapore.  In this session, we will talk about the backend system operation when deploying OpenStack systems(multi locations) on single KeyStone, from a developer’s point of view.

This session will cover:
- Replication of KeyStone databases.
- In the case of the token table, Striping the table in order to reduce replication cost

About GMO Internet, Inc.
GMO Internet Group, headquartered in Tokyo, is a leading force in the Internet industry offering one of the most comperehensive ranges of Internet services worldwide. We are providing a public cloud called “ConoHa” and “GMO APPs Cloud” as part of our services. Both are based on OpenStack.

Naoto Gohko
Naoto-Gohko, GMO Internet, Inc.
our product by OpenStack. | ConoHa public cloud; https//www.conoha.jp/en/ | GMO AppsCloud; https://cloud.gmo.jp/en/

HYUNTAE PARK
Assistant Manager, GMO Internet, Inc.
Hyuntae Park, GMO Internet, Inc. | [email protected] | https://www.linkedin.com/profile/preview?locale=ja_JP&trk=prof-0-sb-preview-primary-button

Naoto Gohko

October 27, 2015
Tweet

More Decks by Naoto Gohko

Other Decks in Technology

Transcript

  1. OpenStack Summit Tokyo Hyuntae Park <[email protected]> Naoto Gohko <[email protected]> Team

    Manger / OpenStack Engineer IT Architect Enginner Multi location implementation of OpenStack
  2. š History of our services using OpenStack š Overview of

    Multi Region š Original extensions to OpenStack š Multi Region supported Domain # Agenda
  3. š Diablo š Grizzly š Havana š Juno V1 V1,V2

    V3 2012 - 2013 - 2015 - V2 IPv6 / GRE on Virtual Network Block Storage Object Storage # History of our services using OpenStack
  4. š Multi Region š SSD Only š Scalability š API

    š Simple and competitive pricing # Newly Released ConoHa
  5. Share single KeyStone in multiple OpenStack environment Keystone Neutron Glance

    Nova Compute node Compute node Block Storage(Cinder) object Storage (swift) Keystone Neutron Glance Nova Compute node Compute node Block Storage(Cinder) object Storage (swift) Keystone Tokyo Region Singapore Region # Description of Multi Region
  6. CentOS 7.1 x86_64 Juno (RDO) Maria DB Connect to Tokyo

    KeyStone from All regions. Add each region endpoints to Tokyo KeyStone. Did not need to modify OpenStack code. š OS and OpenStack Versions š Multi Region Setting # Specs
  7. Tokyo SanJose Singapore API  Management Keystone API API  Management Keystone

    API API  Management Keystone API User/tenant token Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute # Multi Region Design 1
  8. Tokyo Sanjose Singapole User/tenant User/tenant API  Management Keystone API API

     Management Keystone API API  Management Keystone API User/tenant DB Replication DB Replication Nova-API Neutron Glance Cinder Nova-compute Nova-API Neutron Glance Cinder Nova-compute Nova-API Neutron Glance Cinder Nova-compute Token Token Token Tokyo SanJose Singapore User/tenant token User/tenant token API  Management Keystone API API  Management Keystone API API  Management Keystone API User/tenant token DB Replication DB Replication Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute # Multi Region Design 2
  9. # Issues and Restrictions on Multi Region š User-registration is

    possible in Japan only š VPN performance issue š Issues on replicating token table.
  10. Tokyo Singapole User/tenant User/tenant API  Management Keystone API API  Management

    Keystone API API  Management Keystone API Token Token Tokyo SanJose Singapore API  Management Keystone API API  Management Keystone API READ/WRITE READ READ Token Token Token Do not create/delete users Do not create/delete users Our Customer base User administration # User-registration is possible in Japan only DB Replication DB Replication User/tenant User/tenant User/tenant R/W R/W
  11. API  Management Keystone API KeystoneDB Nova Neutron Glance Cinder OpenStack

    Cluster Nova Get/token Glance Get/token Neutron Get/token Cinder Get/token VM Create ! Nova user token:001 Neutron Token:002 Glance Token:003 Cinder Token:004 VM Create ! VM Create ! Nova user token:002 Neutron Token:003 Glance Token:004 Cinder Token:005 Nova user token:006 Neutron Token:007 Glance Token:008 Cinder Token:009 # Bloat access tokens š Too many tokens will be created from each components.
  12. Setting example.conf [keystone_authtoken] token= 100 year expires token [neutron_authtoken] token=

    100 year expires token [glance_authtoken] token= 100 year expires token [cinder_authtoken] token= 100 year expires token # Issues on replicating token table. š 100 year expires token We fixed it so that any tokens can be used for each components.
  13. # Now Feature Singapore Tokyo SanJose User/tenant User/tenant ݱ ࡏ

    Keystone API API  Management Keystone API API  Management Keystone API User/tenant DB Replication DB Replication Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute Token Token Token Tokyo SanJose Singapore User/tenant token User/tenant token API  Management Keystone API API  Management Keystone API API  Management Keystone API User/tenant token DB Replication DB Replication Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute Nova Neutron Glance Cinder Nova-compute
  14. # Replication design Singapore Region Keystone DB(S) MariaDB Galera Cluster

    Keystone DB(M) Keystone DB(S) MariaDB Galera Cluster Keystone DB(M) Keystone DB(S) MariaDB Galera Cluster Keystone DB(M) Tokyo Region SanJose Region MariaDB Replication MariaDB Replication
  15. Tokyo Keystone API Keystone API Keystone API OpenStack Keystone User/tenant

    New Region Token Nova Neutron Glance Cinder OpenStack Component Cluster OpenStack Component VM Template OpenStack Component VM Template Token Nova Neutron Glance Cinder OpenStack Component Cluster SanJose Region Token Nova Neutron Glance Cinder OpenStack Component Cluster User/tenant Tokyo Region Token Nova Neutron Glance Cinder OpenStack Component Cluster Keystone User/tenant User/tenant Singapore Region # How to add new region Keystone Keystone Keystone
  16. # Multi Region can provide... Tokyo Region Glance /var/lib/glance/images/{tenant-id}/example.qcow2 qcow2

    image Symbolic Link http://glance.conoha.io/{uuid}/example.qcow2 Web Share Sanjose Region Glance Glance image download qcow2 image /var/lib/glance/images/{tenant-id}/example.qcow2
  17. # Feature plans Tokyo User/tenant API  Management Keystone API Token

    Singapore OpenStack Cluster Customer Base Create/Delete user API  Management Keystone Nova-Compute on VM Tokyo User/tenant API  Management Keystone API Token Tokyo OpenStack Cluster API  Management Keystone Nova-Compute on VM Tokyo User/tenant API  Management Keystone API Token Sanjose OpenStack Cluster API  Management Keystone Nova-Compute on VM KeystoneDB KeystoneDB Sync All Keystone Sync All Keystone KeystoneDB Personal VM Personal VM Personal VM GMO PaaS GMO PaaS GMO PaaS Customer Base Customer Base Create/Delete user Create/Delete user
  18. 24 Swift  cluster (.0*OUFSOFU *OD714BOE$MPVETFSWJDFT 0OBNBFDPN 714   IUUQXXXPOBNBFTFSWFSDPN

    'PSDVTHMPCBM*1TQSPWJEFECZTJNQMFOPWBOFUXPSL UFOUFO 714  IUUQXXXUFOUFOWO 4IBSFPG044CZ(SPVQDPNQBOJFTJO7JFUOBN $POP)B 714   IUUQXXXDPOPIBKQ 'PSDVT2VBOUBN /FVUSPO PWFSMBZUFOBOUOFUXPSL (.0"QQT$MPVE   IUUQDMPVEHNPKQ 0QFO4UBDL )BWBOBCBTFETU SFHJPO &OUFSQSJTFHSBEF*BB4 XJUICMPDLTUPSBHF PCKFDUTUPSBHF  -#BB4 BOECBSFNFUBM DPNQVUFXBTQSPWJEFE 0OBNBFDPN $MPVE  IUUQXXXPOBNBFDMPVEDPN 'PSDVT-PXQSJDF7.JOTUBODFT CBSFNFUBM DPNQVUFBOEPCKFDUTUPSBHF $POP)B $MPVE  IUUQXXXDPOPIBKQ 'PSDVT.-WYMBO PWFSMBZ -#BB4 CMPDLTUPSBHF %/4BB4 %FTJHOBUF BOEPSJHJOBMTFSWJDFTCZLFZTUPOFBVUI OpenStack Diablo on  CentOS 6.x Nova Keystone Glance Nova  network Shared  codes Quantam OpenStack Glizzly on  Ubuntu  12.04 Nova Keystone Glance OpenStack Havana on  CentOS 6.x Keystone Glance Cinder Swift Swift Shared  cluster Shared  codes Keystone Glance Neutron Nova Swift Baremetal compute Nova Ceilometer Baremetal compute Neutron LBaaS ovs +  gre tunnel  overlay Ceilometer Designate Swift OpenStack Juno on  CentOS 7.x Nova Keystone Glance Cinder Ceilometer Neutron LBaaS (.0"QQT$MPVE   IUUQDMPVEHNPKQ OE SFHJPOCZ0QFO4UBDL +VOPCBTFE &OUFSQSJTFHSBEF*BB4 XJUI)JHI*014*SPOJD$PNQVUFBOE/FVUSPO-#BB4 Upgrade   Juno GSLB Swift Keystone Glance Cinder Ceilometer Nova Neutron Ironic LBaaS
  19. 26

  20. 5IFNFBOJOHPGUIFXPSE • %PNBJO • ,FZTUPOFEPNBJO • 8JUIW"1*TFSWJDF PVSDMPVE • %/4%PNBJO

    • -PDBUJPO • %JGGFSFOUHFPHSBQIJDMPDBUJPOTPOUIF&BSUI • 64 4BO+PTF +1 5PLZP 4( 4JOHBQPSF • 3FHJPO • 0QFO4UBDL SFHJPO • -PDBUJPO3FHJPO • $BOTFUVQVQNVMUJQMF3FHJPO JOPOF-PDBUJPO
  21. $POP)BCBTFEPO0QFO4UBDL +VOP *BB4  • .VMUJQMFSFHJPOPQFOTUBDL DMVTUFS • 5PLZP4JOHBQPSF 4BO+PTF

    • BOETPPO • 'VMM44%TUPSBHF • .VMUJQMFLFZTUPOFTFSWJDFEPNBJOTVQQPSU • $POP)B BOE/FYUTFSWJDF OPXJOEFWFMPQNFOU 0&.FUD • -#BTB4FSWJDF-74%43 PSJHJOBM • %/4BTBTFSWJDF0QFO4UBDL %FTJHOBUF • 0QFO4UBDL "1*BOEBEEJUJPOBM3&45GVM "1* • .VMUJQMF-BOHVBHFTXFCQBOFMTVQQPSU • +BQBOFTF $POP)B &OHMJTI  ,PSFBO .BOEBSJO$IJOFTF
  22. 29 $POP)BCBTFEPO0QFO4UBDL +VOP 4BB4 • 0SJHJOBMTIBSFETFSWJDFT 1BB44BB4 CZLFZTUPOFBVUI – &NBJMIPTUJOHBTBTFSWJDF

    – %BUBCBTFIPTUJOHBTBTFSWJDF – 4IBSFEXFCIPTUJOH $1BOFM BTBTFSWJDF – 8PSEQSFTT IPTUJOHBTBTFSWJDF è PO0QFO4UBDL 8JUIEPNBJOTVQQPSU
  23. 30 • 0QFO4UBDL %FTJHOQBUUFSO4FSWJDF – &OUFSQSJTFHSBEFXIJDIJTBQQSPQSJBUFGPSUIFQSJDF • /FUXPSLEFWJDF --##SPDBEF"%9WT-74%43 •

    4UPSBHF%FWJDF CSPDL)11"3 /FU"QQWT4%4 [GT  – /FUXPSLNPEFMT /"5 'MBU FUD • .--JOVY#SJEHFWT.-PWT • 8FBSFPQFSBUJOHTFWFSBM0QFO4UBDLDMVTUFSGPSEJGGFSFOUQSPEVDT – *UIBTBPQUJNJ[FEGPSFBDIQSPEVDUT 3FBTPOPG0QFO4UBDL*BB41SPEVDU
  24. 31 • 0QFO4UBDL – *OUFHSBUJPOPG0QFO4UBDLDMVTUFS – 8JUIUIFOFXMZBGGPSEBCMF44% 5IFTUPSBHFEFWJDF CFDPNFTNPSFIJHIQFSGPSNBODFXJUIMPXFSDPTUT –

    8JUI.-OFVUSPOTVQQPSU .VMUJOFUXPSLNPEFMXJMMCF BWBJMBCMF – *UJTQPTTJCMFUPEFWFMPQPVSQSPEVDUTGPSFBDIDPVOUSZXJUI .VMUJ3FHJPOTVQQPSUBOE.VMUJ-PDBUJPO TVQQPSU è 8FDBOJOUFHSBUFPVS0QFO4UBDLDMVTUFS (SPXPVUPG0QFO4UBDL*BB41SPEVDU
  25. 32 • %POPUXBOUUPVTF,FZTUPOF7"1* – .BOZDMJFOUTEPFTOPUTVQQPSUJU • .JOJNBM,FZTUPOF NPEJGJDBUJPO • .VMUJEPNBJOTDPQFTUSVDUVSFE

    • $PNNPODMPVESFTPVSDF – $PNQVUF 4UPSBHF /FUXPSL %/4 1BB4 • *OEFQFOEFOU*UFNTGMBWPS WPMVNFUZQF JNBHFT – $PNQBUJCMFEPNBJOTDPQFBOE %PNBJOTXJUI7"1*PO0QFO4UBDL+VOP .VMUJ%PNBJOLFZTUPOF
  26. 33 • 8FVTFBOEDVTUPNJ[FUIFDPEFUIBUJTJO+VOP,FZTUPOFW EPNBJO – &OBCMF%PNBJO*%GPS+VOP,FZTUPOF7"1* • 4BB4JNQMFNFOUBUJPOXJUIQZUIPOLFZTUPOFDMJFOU – 1SPDFTTSFMBUFE%PNBJO*%

    BOE%BUBJNQMFNFOUBUJPO è%PNBJO*%GSPNUPLFO"1* 6TFS 1045WUPLFO "ENJO TFSWJDF  (&5WUPLFO\JE^ +VOP,FZTUPOF7"1*%PFTOPUTVQQPSU%PNBJOT
  27. 34 %PNBJOTBOEVTFSQSFGJYOBNFTQBDF ,FZTUPOFXSBQQFSQSPYZBUEPNBJOTQFDJGJDLFZTUPOFFOEQPJOU Domain Product Prefix name  space gnc ConoHa

    gnc zjp JP  OEM-­‐1 zjp zsg SG  OEM-­‐ 1 zsg ...  ...   OEM-­‐n ...  ... &YQ VTFSHOD *NBHFOBNFHOD@DFOUPT
  28. 36 Keystone DB user/tenant token Geographic endpoints Default  domain (Admin

     only) (services_auth_url) ConoHa public  API gncu72074485   …. ConoHa web  panel  API gncu72074485   …. Openstack Component nova cinder neutron glance glance ConoHa Dashboard gncu-­‐vpsadmin Sync  (region  – region) Admin  &   Internal   endpoint Z.Com domain (admin_auth_url) Public  endpint Z.Com domain (user_auth_url) ※openstack original   Endpoint Admin  &  Internal   Endpoint gnc domain (user_auth_url) Public  Endpoint gnc domain (admin_auth_url) ConoHa domain(gnc) Z.Com domain "EEJOHEPNBJO OE DMPVE[DPN BOE$POP)B FOEQJOU z.com public  API zjpu82073435   ….
  29. 40 è %PNBJOTQFDJGJD*UFNT/POF • /FVUSPO.-VTJOHNVMUJQMF/FUXPSLQMVHJOT – PWT79-"/.-GPS*BB4 TFSWJDF – -JOVY#SJEHF.-GPS1BB44BB4

    • -#BB4-74%43ESJWFS PSJHJOBM – 8JUIPVUEFQFOEPOEPNBJODPOGJH • 1VCMJDOFUXPSLDPOGJH – 8JUIPVUEFQFOEPOEPNBJODPOGJH – 'MBU/FUXPSLNPEFMCZ'JY*1XJUIPVUGMPBUJOH*1 /FVUSPO/FUXPSLJOH
  30. 41 è %PNBJOTQFDJGJD*UFNT /PUSFRVJSFE – 40"SFDPSE – /4SFDPSE • %JGGFSFOU[POFTJTSFHJTUFEUP%/4

    Example) Domain “gnc”: conoha.io [n-gohko@corpscan01~]$ dig +short plastic-machine.red ns ns-a3.conoha.io. ns-a1.conoha.io. ns-a2.conoha.io. [n-gohko@corpscan01~]$ dig +short plastic-machine.red soa ns-a1.conoha.io. postmaster.example.org. 1441328665 3600 600 86400 3600 %FTJHOBUF%/4
  31. 44 è%PNBJOTQFDJGJDJUFNT – 1VCMJDJNBHFT • 6TFSTDBOTFFUIFJNBHFUIBUCFMPOHUPEFGBVMUEPNBJOPSUIFJS EPNBJO • -JTUPG"1*(&5WJNBHFT –

    %FGBVMU%PNBJOTIBSFEQVCMJDJNBHF B – %PNBJO HOD [KQ FUD %PNBJOTQFDJGJDQVCMJDJNBHF C   B • 3FDPSE%PNBJO*%JOUPUIFNFUBEBUBPGQVCMJD*NBHF è -JTUPQFSBUJPOXBTDVTUPNJ[FECZ"1*XSBQQFSQSPYZ (MBODF*NBHFTFSWJDF Public  images  (a) Domain  Public  images  (b)
  32. 48 è%PNBJOTQFDJGJDJUFNT – SaaS service endpoint (SMTP/POP/IMAP/DB/CPanel) • Mail hosting

    service • Database hosting service • Web(Cpanel) hosting service • Wordpress hosting service – Domain specific: none Request for KeyStone Admin endpoint(ex. GET /v2.0/token/{tokenid}) need to access the Domain side endpoint 0SJHJOBMTPGUXBSFBTBTFSWJDFVTJOHLFZTUPOFBVUI
  33. 51 .PUJWBUJPO • $SFBUFTDPQFJOUIFEPNBJO – 4DPQFEJUFNT • 'MBWPS • *NBHFT

    • 7PMVNFUZQF – 4IBSFEJUFNT • 1VCMJD/FUXPSLT • )ZQFSWJTPS • *NBHFT %FGBVMUEPNBJO • 6TJOH,FZTUPOF"1*W
  34. 52 Keystone DB regionTYO1 OSA1 SIN1 SJC1 %PNBJOQSPEVDUEPNBJO SaaS IaaS

    SaaS IaaS SaaS IaaS SaaS IaaS user/tenant user/tenant user/tenant token token token sync sync Geographic endpoints ConoHa keystonea pi ConoHa keystone   api ConoHa keystone   api JP (auth_url) SG (auth_url) US (auth_url) ConoHa
  35. 53 Keystone DB regionTYO1 OSA1 SIN1 SJC1 SaaS IaaS SaaS

    IaaS SaaS IaaS SaaS IaaS user/tenant user/tenant user/tenant token token token sync sync Geographic endpoints ConoHa keystonea pi ConoHa keystone   api ConoHa keystone   api JP (auth_url) SG (auth_url) US (auth_url) ConoHa JP  OEM-­‐1 JP  OEM-­‐1 keystonea pi JP  OEM-­1 (auth_url) %PNBJOQSPEVDUEPNBJOT
  36. 54 %PNBJOQSPEVDUEPNBJOT Keystone DB regionTYO1 OSA1 SIN1 SJC1 SaaS IaaS

    SaaS IaaS SaaS IaaS SaaS IaaS user/tenant user/tenant user/tenant token token token sync sync Geographic endpoints ConoHa keystonea pi ConoHa keystone   api ConoHa keystone   api JP (auth_url) SG (auth_url) US (auth_url) ConoHa JP  OEM-­‐1 SaaS  only  OEM-­‐2 SaaS  only  OEM-­‐2 SaaS  only  OEM-­‐2 SaaS  only  OEM-­‐2 JP  OEM-­‐1 keystonea pi JP  OEM-­1 (auth_url) SaaS  only   OEM-­‐2 keystonea pi SaaS  only   OEM-­‐2 keystonea pi SaaS  only   OEM-­‐2 keystonea pi SaaS  only OEM-­2 (auth_url) SaaS  only OEM-­2 (auth_url) SaaS  only OEM-­2 (auth_url)
  37. 55 Keystone DB regionTYO1 OSA1 paas vps paas vps user/tenant

    token Geographic endpoints JP ConoHa keystonea pi nova ▲api ・db ・scheduler ・novncproxy ・conductor ConoHa client 1)token  get 2)return  token & Endpint list 3)req:  create  VM 4)return:  vm created &generate  VM  ID z.com keystonea pi (auth_url) (auth_url)
  38. 56 Keystone DB regionTYO1 OSA1 user/tenant token Geographic endpoints ConoHa

    keystonea pi ConoHa Client Z.com Client Z.com keystonea pi 1)token  GET 2)return &エンドポイントリスト 3)VM作成の要求 4)VM作成&VMのID発行 JP (auth_url) (auth_url) paas vps paas vps nova ▲api ・db ・scheduler ・novncproxy ・conductor nova ▲api ・db ・scheduler ・novncproxy ・conductor
  39. 57 • 8FIBWFCFFOPQFSBUJPOUIFTFSWJDF *OUIF.VMUJ-PDBUJPO UP CVJMEB.VMUJ3FHJPOFOWJSPONFOU • *OPSEFSUPCFUUFSUBLFBEWBOUBHFPGUIF-PDBUJPOBOE3FHJPO  XFBQQMZUIFDPODFQUPG%PNBJOUPNBLFUIFFOEQPJOUBOEUIF

    "1*XSBQQFSQSPYZ • EPNBJO@JE YYY@EPNBJO@JE • ESJWFSLFZTUPOFDBUBMPHCBDLFOETUFNQMBUFE$BUBMPH • UFNQMBUF@GJMFFUDLFZTUPOFEFGBVMU@DBUBMPHUFNQMBUFT • "/%XSBQQFS QIQ SPVUFSQSPHSBNT 8FVTFJUUPDVTUPNJ[FBTNVMUJQMF*BB4J 'JOBMMZ