OpenStack Preemptible Instance

OpenStack Preemptible Instance

5fccdc4e847bd1a11aea2cea64cca752?s=128

Tetsuro Nakamura

July 23, 2019
Tweet

Transcript

  1. 1.

    Copyright©2019 NTT corp. All Rights Reserved. OpenStack Preemptible Instance OpenStack

    Days Tokyo 2019     https://speakerdeck.com/tetsuro0907/openstack-preemptible-instance
  2. 2.

    1 Copyright©2019 NTT corp. All Rights Reserved.   

    : Tetsuro Nakamura (@tetsuro) NTT +'*24 "4-$#$(0 • 74/3%4$%.* ü DPDK 5SPP7 )4&,$6 ü OpenStack 5PlacementBlazar6 • 7 ü 1!-4
  3. 3.

    2 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack  3. Preemptible instance 4. Part Time Reservation 5.  
  4. 4.

    3 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack  3. Preemptible instance 4. Part Time Reservation 5.  
  5. 5.

    4 Copyright©2019 NTT corp. All Rights Reserved. 1. HFB&#/-3'!6< 371,6HFB<

    +:(3" 2. 8383PTIHFB< 9%:AOUXJ=WCEFJQ6.&!BR?M6AXDFLUX H5,6& );00#:" A) Preemptible Instance: $ 6PEW<KNV>+:-76SGXF&94$!SGXF<+: -7557:(362':6PEW6(3" B) Kubernetes PreemptionYv1.9-Z: BRFH6SGXF&*-5!5 &$ Pod <$ *1 & $ Pod <+:" Preemption   HFBB HFBB HFBA NS@WNL
  6. 6.

    5 Copyright©2019 NTT corp. All Rights Reserved. Preemptible Instance •

    An]od^mVqs`psi\),UtUu8 Ø .:0tbjm9%B(*:0/u Ø lfagsd9%tWeb`psnrau Ø kfe * etc. 1. Google Compute PlatformtPreemptible VMu i. ;V3 V1V<Q +6 ii. FOQW 2K[Z6HEZ t!QW24 >Q 2u 2. Amazon Web ServicestSpot Instanceu i. +MZ^rcdrcV +&HLPGYB7 H $LN&\ +&H ZR ,U^rcdrcH@ ii. ^rcdrcH +=-U@K[ZV\?JC34#>V1DXBAuto Scaling\ +QIZCn`_chR5DTSV'6"
  7. 9.

    8 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack  Ø Placement Project Ø Blazar Project 3. Preemptible instance 4. Part Time Reservation 5.  
  8. 10.

    9 Copyright©2019 NTT corp. All Rights Reserved. Placement 1. &98:OFRD5

    <(JPC?@I 2. 5SOpenStack4!93'T *;-AQEJI</ A) Resource ProviderU OFRD5 B) Resource Class: OFRD5# C) Inventory: OFRD5 D) Consumer: OFRD5 3.  $Nova)Placement53@N>=QI A) NovaU MBQ5HJP><(JPC?@I B) LDI5 4Placement<':% OFRD VHRGKRD5 S72726Nova54&.-)$SQL)" ,:51REST-IF<.0Nova2 4+-T
  9. 11.

    10 Copyright©2019 NTT corp. All Rights Reserved. Nova-Placement Interaction 1.

    OBH.+4“nova-compute”( /Placement2TDXB6 Ø I9KJ9@0  Ø libvirt (KVM, QEMU etc.), Xen Server, Hyper-V etc. 2. VM0"1#VM0 BNG=6Nova'3Placement/)-OBH6 4 3. Nova0 L79U/ *5,*3/'%L8UETW>1Nova0B?AQXSMVCB Y"nova-scheduler"Z.&$ nova-api nova- conductor nova- scheduler placement RPC REST nova- compute TDXB OBH B<XPF:G= BNG= 9WBEWB OBH0! RX@T=;BH
  10. 12.

    11 Copyright©2019 NTT corp. All Rights Reserved. Placement Extraction 1.

    Newton-Rocky L Placement in Nova A) <KC'Nova/HF>BH,#AK@EK?2Nova B) Nova-/IF0REST,7+$6'#DI>9;B 0' " 2. Stein- L Placement DI>9;B A) HF>BH#AK@EK?%41DI>9;B/ 8Nova&5. B) DI>9;B/ .46 3 /' C) Nova/Placement<KC0SteinHHK?,0 Ø TrainHHK?&5!(7*Placement8)6:D=GJ2 
  11. 13.

    12 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack  Ø Placement Project Ø Blazar Project 3. Preemptible instance 4. Part Time Reservation 5.  
  12. 14.

    13 Copyright©2019 NTT corp. All Rights Reserved. Blazar 1. OFZMHL\d_$D

    +=82;:-<^NdM>&eh^NdM fD#0 2. Nova@Neutron;*4>^NdM> DIXdR. A) FcMOcM i. WMR>^NdM>+> Ø  ,=YKc19ACBe7?>^NdM1%3CBf2;D& ii. WMR>^NdM> 9D eWMR  !f Ø  47]dJ>' >YKc1 5=QUaF3C</2;D&6B B) TPRbdG g i. Floating IPES`M 3. -Ironic;*49-VE[O_> DIXdR6B.e(f A) ]dMHdMg FcMOcMD9B >):"=VE[O_D0.
  13. 15.

    14 Copyright©2019 NTT corp. All Rights Reserved. User :10000 Feet

    Overview Reservation ID Reservation ID Reservation ID Reservation ID   3+7  ID !#  ID% 8 9  2,/  3+7%5*0-4   3+7  2,/  2,/  "'$2,/& 3+7(.16)  Reservation ID
  14. 16.

    15 Copyright©2019 NTT corp. All Rights Reserved. Blazar – Ironic

    Usecase • IronicO – F9M%+(F9M1 OpenStack*CL:45= – F9M% OS('11D2G<K&-!.D2G<KCLB:I?M 61/8MEN@M=& – D2G<K1VM& ) /!.*Nova*>J3A"#!)$ ”Ironic stand alone” &,0/HN;7N;  • Nova Ironic driver • Ironic stand alone OS Container HW OS libvirt VM HW OS HW
  15. 17.

    16 Copyright©2019 NTT corp. All Rights Reserved. User @10000 Feet

    Overview Reservation ID  5,81:  ID $  90=/    (6/3  Ironic stand alone'!"*5,81:  5,81:24;-  90=/   #+%7.<)5,81:24;-  ID&6/3 > ? Reservation ID  Reservation ID
  16. 18.

    17 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack   3. Preemptible instance Ø Nova 4. Part Time Reservation   5. 
  17. 19.

    18 Copyright©2019 NTT corp. All Rights Reserved. PLACEMENT cell1 Preemtible

    OpenStack Preemptible Instance • ,8941*# ?Rocky- : @ 1.  7-=26<+NoValidHost)&Nova  7-=)Pending State" 2. Reaper()Preemptible Instance)%'! :/>.) 3. :/>. " & ReaperPending State#+=.0=.):5;3 ?$+=.0=.)Error State"@ ? REAPER 1. VM create HW OS HW OS VM VM cell0 Failed VM HW OS VM VM 3. Error State Pending State 2. Delete 2. NoValidHost 1. Detect 3 Rebuild Preemtible
  18. 20.

    19 Copyright©2019 NTT corp. All Rights Reserved. • 4<>:71, 

    • Nova+ A • Introduce Pending VM stateA • https://review.opendev.org/#/c/648687/ • Enable Rebuild for Instances in cell0A • https://review.opendev.org/#/c/648686 • Nova,+" %" A • Reaper FunctionA • https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype • A • Reaper '$'$!'OpenStack+ %";?52380  0B =@79@6- /0,B • * *"Preemptible Instance &#(0$).0 OpenStack Preemptible Instance ? REAPER
  19. 21.

    20 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance 2. OpenStack   3. Preemptible instance 4. Part Time Reservation  Ø OpenStack Blazar 5. 
  20. 22.

    21 Copyright©2019 NTT corp. All Rights Reserved. Proposal 1. :Blazar9.1>ASGISG!VFull

    Time Reservation Ø OTD/!'*"6#8QHTG&? : ,7/24. 9:00 - 7/27. 9:00- 5 ,4C@ 16GB LES 1 -?! ⇒ Blazar/!: * ?% 2. 3>48Blazar:ASGISG!V Part Time Reservation Ø OTD/JTIR:(?20 V,7/24. 9:00 - 7/31. 9:00- :(5JTIR5,72(-* ,4C@ 16GB LES 1 -?! ⇒ !: ;*Blazar/!?%3+ )Q@RIAM@KQBTEPS/$: *QHTG9<=* !(?24=*QGBFNTR2>+ ,7/25 . 9:00 - 7/26 9:00-U,7/28 . 9:00 - 7/30 9:00-87+ Blazar Part Time Reservation Feature
  21. 23.

    22 Copyright©2019 NTT corp. All Rights Reserved. Preemtible Instance Part

    Time Instance Part Time Reservation – Work Flow Preemtible Instance Preemtible Instance  ,   … ,      ,       Part Time Instance     Part Time Instance • Preemptible Instance • Part Time Reservation 
  22. 24.

    23 Copyright©2019 NTT corp. All Rights Reserved. 1. Preemptible Instance62BLT5V5RG2IH<0Part

    Time ReservationDJLAC &2oq[nqfW HeqdmI#-IM) A) Preemptible Instancew Ø rjq_s VM6K<21"($GIDLACOQ21 Ø r[kZfs=94L=602BLT5JV5SL>X1 B) Part Time Reservationw Ø rjq_sVM6K<21"($GIDLACOQ260*:LDHv- 5<@21 Ø r[kZfsDJ0+Ht-E0Hu-0 5<L=1 2. Part Time ReservationI w / 5=eqdmI#-6 5R);UT1 / LA@g`pI ,Wjq_DPT-6!:T1 / BlazarIb]aiqlp\Ym^lchID=9'D7?31 . HNC&2E23 %DJG80;KFHJD7G21 Part Time Reservation - Summary
  23. 25.

    24 Copyright©2019 NTT corp. All Rights Reserved.  1. Preemptible

    instance  2. OpenStack    3. Preemptible instance  4.
  24. 26.

    25 Copyright©2019 NTT corp. All Rights Reserved. /*0 • Blazar*Placement,'(

    • Nova) !2(1Preemptible Instance3 • Blazar)+Part Time Reservation 3 "2."/#?$-RFE@Request for EnhancementA 2.4576;>:=- -upstream developer@*A 3&( %!@-<983/+)?A
  25. 27.

    26 Copyright©2019 NTT corp. All Rights Reserved.  ?7-=452.6<3 A)

    Part Time Reservation ':=0/=0? Ø GCP$AWS*## +!@ Ø Spot Instance !@ Ø Part Time Reservation %@ Ø Preemptible Instance@ Ø ")Reaper*Blazar/Tacker&&+!@ B) 8,91;  ':=0/=0? Ø Ironic *## +!@ Ø virt driverNova !@stand alone@ Ø Baremetal Reservation %@ C)  ( openstack-jp-ops@slack.com  @tetsuro !>
  26. 28.

    27 Copyright©2019 NTT corp. All Rights Reserved. 1. OpenStack Placement:

    A) 756@DAC4%  -,  E i. SteinAAD7!%:>B2?7;% : Ø https://review.opendev.org/#/c/646029/ ii. RockyAAD7!%<4=1937$+. : Ø https://review.opendev.org/#/c/659207/ B) 0##756@DAC4E"/& 'E 2. DPDK SPP: A) DPDK>B83;% ) ' ( E Ø http://git.dpdk.org/apps/spp/ Ø https://www.youtube.com/watch?v=RIOHSPrvyXU  F0#")* '
  27. 30.

    29 Copyright©2019 NTT corp. All Rights Reserved. : Extraction Step.

    1. "Placement5H=(:1GFH=$0G7<HD 2. Placement >.0D( A) ``nova.conf``([placement]836AG(``placement.conf``*( B) ``placement-policy.yaml``(placement;/E3<C*( 3. Placement(;H9?H7( 4. ;H9?H7@04EH6AG(nova-api -> placement) A) mysql-migrate-db.sh, postgre-migrate-db.sh 5. ;H9?H7 (ICCH7 J A) 2>B0G@04EH6AGK CCH7#"! '&,(&% B) 2GB0G@04EH6AGK TrainCCH7+Extracted Placement- )2>B0G#   https://docs.openstack.org/placement/latest/upgrade/to-stein.html