The Continuity of Out-of-band Remote Management Across Virtual Machine Migration in Clouds

In Infrastructure-as-a-Service (IaaS) clouds, users remotely manage the systems in virtual machines (VMs) called user VMs, e.g., through VNC. To allow users to manage their VMs even on failures inside the VMs, IaaS usually provides out-of-band remote management. This VM management is performed indirectly via a VNC server in a privileged VM called the management VM.

However, it is discontinued when a user VM is migrated from a source to a destination host. This is because a VNC server in the management VM at a source host is terminated on VM migration. Even worse, pending data is lost between a VNC client and a user VM.

In this slide, we propose D-MORE for continuing out-of-band remote management across VM migration. D-MORE provides a privileged and migratable VM called DomR and performs out-of-band remote management of a user VM via DomR. On VM migration, it synchronously co-migrates DomR and its target VM and transparently maintains the connections between a VNC client, DomR, and its target VM.

We have implemented D-MORE in Xen and confirmed that a remote user could manage his VM via DomR even after the VM has been migrated. Our experiments showed that input data was not lost during VM migration and the overhead of D-MORE was acceptable.

Transcript

1. © Sho KAWAHARA, KSL, Kyushu Institute of Technology The Continuity of Out-of-band Remote Management Across Virtual Machine Migration in Clouds Sho Kawahara and Kenichi Kourai Kyushu Institute of Technology, Japan
2. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Remote VM Management in IaaS VM VM VM VM Management VNC  In-band remote management is usually used ◦ A VNC server runs in a user VM ◦ A user connects to the server via the network  However, a user cannot access his VM when ◦ He fails the configuration of the firewall or network ◦ The system in the VM does not work normally VNC Client VNC Server
3. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  Access a user VM via a VNC server in the management VM ◦ Directly access virtual devices for a user VM  Virtual keyboard, mouse, and video card ◦ Not rely on the network or system in the user VM Out-of-band Remote Management VNC Client Management VM VNC Server Virtual Devices User VM Network Failure Out-of-band Remote Management In-band remote management
4. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  VM management is discontinued on VM migration ◦ A VNC server at a source host is terminated  It loses the access to the removed virtual devices  A user has to manually reconnect to the destination ◦ After identifying the reason and looking for the destination host Discontinued on VM Migration Management VM VNC Client VNC Server User VM Migration Management VM ？ Virtual Devices VNC Server Virtual Devices
5. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  Keyboard and mouse inputs can be lost on VM migration ◦ In-flight data is dropped and is not retransmitted ◦ Pending data in a VNC server is lost by its termination ◦ Pending data in virtual devices is lost by their removals Data Loss on VM Migration Management VM VNC Client User VM Migration Management VM ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD Sending ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD VNC Server VNC Server Virtual Devices Virtual Devices ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD Processing Processing
6. © Sho KAWAHARA, KSL, Kyushu Institute of Technology VNC Server  Continue out-of-band remote management across VM migration ◦ Provide a privileged and migratable VM (DomR)  A VNC server and virtual devices run in DomR ◦ Synchronously co-migrate DomR and its target VM ◦ Maintain the connections between a VNC client, DomR, and its target VM D-MORE Migration DomR VNC Server Virtual Devices User VM Management VM VNC Client Maintained by D-MORE Maintained at the network level
7. © Sho KAWAHARA, KSL, Kyushu Institute of Technology VNC Client  No input data is lost during co-migration ◦ Pending data in a VNC server and virtual devices  Migrated as a part of DomR ◦ In-flight data sent by a VNC client  It is retransmitted because the TCP connection to DomR is maintained Data Loss Prevention DomR VNC Server Virtual Devices Target VM ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD Processing ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD Processing ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD Sending
8. © Sho KAWAHARA, KSL, Kyushu Institute of Technology DomR  A VM with privileges for accessing only its target VM ◦ Establish shared memory  By mapping target VM's memory  Exchange input and output data ◦ Establish interrupt channels  By interception  Notify each other of new data in the shared memory Virtual Devices VNC Server DomR Target VM Interrupt Channels Shared Memory ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD ABCD interrupt
9. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Migration Process of a Single VM  Transfer VM's memory to a destination host with the VM running ◦ Transfer the entire memory ◦ Repeat to transfer dirty memory until small enough  Stop the VM and transfer the remains ◦ Still dirty memory and CPU state  Restart the VM at the destination stop source host destination host create restart terminate original VM new VM memory transfer
10. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Co-migration  Synchronize two migration processes of DomR and its target VM ◦ Stop the two VMs simultaneously to reduce downtime ◦ Restore shared memory after reconstructing target VM's memory ◦ Save/Restore interrupt channels while stopping the VMs stop DomR Target VM source host destination host create terminate DomR Target VM restore share mem restart save channels restore channels
11. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Restoring Shared Memory  Save the mapping state of target VM's memory ◦ Inspect the page tables of DomR ◦ Set a monitor bit in a PTE if target VM's page is mapped  Meaning shared memory  Remap target VM's memory at the destination ◦ Inspect received page tables ◦ Restore the mapping state if a monitor bit is set in a PTE DomR Target VM map (share) remap (share) page table source destination
12. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Restoring Interrupt Channels  Save the state of interrupt channels ◦ Obtain a list of the interrupt channels  Only between DomR and its target VM ◦ Transfer pairs of ports used for the interrupt channels  Re-establish interrupt channels at the destination ◦ Guarantee that the same port numbers are used ◦ We have modified the resume operation in the OS DomR Target VM Interrupt Channels Interrupt Channels port:2 port:10 port:2 port:10 source destination
13. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Data Transfer in Shared Memory  DomR's writes to shared memory are not detected ◦ Only target VM's writes are detectable ◦ Data written by DomR may not be transferred  D-MORE always considers shared memory as dirty ◦ Guarantee that updated shared memory is transferred DomR Target VM share ？ detect dirty
14. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  We have implemented D-MORE in Xen and Linux  We conducted several experiments for D-MORE ◦ Data loss on co-migration ◦ Performance of remote management and co-migration Experiments Server CPU Intel Xeon E3-1270 3.40GHz Memory DomR 128MB User VM 2GB NIC Gigabit Ethernet VMM Xen 4.3.2 OS Linux 3.7.10 Client CPU Intel Xeon E5-1620 3.60GHz Memory 8GB NIC Gigabit Ethernet VNC Client TigerVNC OS Windows 7
15. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  We examined if D-MORE could prevent data loss during VM migration ◦ Sent a key every 50 ms from a VNC client to a VNC server ◦ Counted the number of lost keys in a virtual keyboard  Original Xen ◦ 1.4 keys were lost  D-MORE ◦ No keys were lost across the co-migration ◦ The number of TCP retransmission was 8.5 Data Loss on Co-migration
16. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  We measured the response time in out-of-band remote management ◦ Keyboard input  We sent a keyboard event and received its remote echo ◦ Full-screen update  We ran a screen saver that redrew the full screen  The overhead was negligible ◦ Keyboard input  Increased by 221 μs ◦ Screen update  Increased by 9 μs Overhead of Using DomR 0 20 40 60 80 100 120 keyboard screen ｒesponse ｔime (msec) original D-MORE
17. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Co-migration Time  We measured the time needed for co-migration of DomR and its target VM ◦ With and without remote management ◦ Independent migration of two normal VMs (baseline)  Proportional to the memory size  Without inputs ◦ Increased by 1.7 s  With inputs ◦ Increased by 15 s ◦ Due to dirty memory 0 5 10 15 20 25 30 35 40 0 512 1024 1536 2048 co-migration time (sec) memory size of the target VM (MB) co-migration co-migration w/o inputs independent migration
18. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  The downtime of DomR and its target VM ◦ We measured the time in which a VM was not running ◦ The downtime of DomR was 3x longer  The user-perceived downtime at a VNC client ◦ We sent a key every 50 ms and measured the response time ◦ The downtime was acceptable Downtime during Co-migration 0 200 400 600 800 1000 1200 0 512 1024 1536 2048 downtime (msec) memory size of the target VM (MB) DomR Target VM VNC downtime
19. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  We examined the impact by co-migration ◦ The response time increased by 5.4 ms  Lasted for 30s ◦ The frame rate decreased by 0.4 fps  Lasted for 30s Performance Degradation by Co-migration 0 100 200 300 400 500 600 700 800 0 20 40 60 80 100 response time (msec) elapsed time (sec) 0 2 4 6 8 10 12 0 20 40 60 80 100 frame rate (fps) elapsed time (sec) Frame rate (Screen) Response Time (Keyboard)
20. © Sho KAWAHARA, KSL, Kyushu Institute of Technology Related Work  VNC Proxy [CloudStack, etc.] ◦ Transparently switch a VNC server on VM migration ◦ Pending data in a VNC server is lost  SPICE [Red Hat, Inc. ‘09] ◦ Support VM migration at the protocol level ◦ VM management depends on specific software  VM Coupler [Kourai et al. ‘13] ◦ Run intrusion detection systems in a dedicated VM (DomM) ◦ Enable co-migration of two VMs for secure monitoring ◦ The synchronization is different from D-MORE
21. © Sho KAWAHARA, KSL, Kyushu Institute of Technology  We proposed D-MORE for continuing out-of-band remote management across VM migration ◦ Run a VNC server and virtual devices in DomR ◦ Synchronously co-migrate DomR with its target VM  Prevent data loss  Reduce downtime  Future work ◦ Apply D-MORE to other remote management software such as SSH ◦ Support fully virtualized guest OSes such as Windows Conclusion