1                                IOEMU stubdom
2                                =============
3
4  This boosts HVM performance by putting ioemu in its own lightweight domain.
5
6General Configuration
7=====================
8
9Due to a race between the creation of the IOEMU stubdomain itself and allocation
10of video memory for the HVM domain, you need to avoid the need for ballooning,
11by using the hypervisor dom0_mem= option for instance.
12
13Using with XL
14-------------
15
16The enable IOEMU stub domains set the following in your domain
17config:
18
19    device_model_stubdomain_override = 1
20
21See xl.cfg(5) for more details of the xl domain configuration syntax
22and https://wiki.xen.org/wiki/Device_Model_Stub_Domains for more
23information on device model stub domains
24
25
26Toolstack to MiniOS ioemu stubdomain protocol
27---------------------------------------------
28
29This section describe communication protocol between toolstack and
30qemu-traditional running in MiniOS stubdomain. The protocol include
31expectations of both qemu and stubdomain itself.
32
33Setup (done by toolstack, expected by stubdomain):
34 - Block devices for target domain are connected as PV disks to stubdomain,
35   according to configuration order, starting with xvda
36 - Network devices for target domain are connected as PV nics to stubdomain,
37   according to configuration order, starting with 0
38 - if graphics output is expected, VFB and VKB devices are set for stubdomain
39   (its backend is responsible for exposing them using appropriate protocol
40   like VNC or Spice)
41 - other target domain's devices are not connected at this point to stubdomain
42   (may be hot-plugged later)
43 - QEMU command line (space separated arguments) is stored in
44   /vm/<target-uuid>/image/dmargs xenstore path
45 - target domain id is stored in /local/domain/<stubdom-id>/target xenstore path
46?? - bios type is stored in /local/domain/<target-id>/hvmloader/bios
47 - stubdomain's console 0 is connected to qemu log file
48 - stubdomain's console 1 is connected to qemu save file (for saving state)
49 - stubdomain's console 2 is connected to qemu save file (for restoring state)
50 - next consoles are connected according to target guest's serial console configuration
51
52Startup:
531. PV stubdomain is started with ioemu-stubdom.gz kernel and no initrd
542. stubdomain initialize relevant devices
553. stubdomain signal readiness by writing "running" to /local/domain/<stubdom-id>/device-model/<target-id>/state xenstore path
564. now stubdomain is considered running
57
58Runtime control (hotplug etc):
59Toolstack can issue command through xenstore. The sequence is (from toolstack POV):
601. Write parameter to /local/domain/<stubdom-id>/device-model/<target-id>/parameter.
612. Write command to /local/domain/<stubdom-id>/device-model/<target-id>/command.
623. Wait for command result in /local/domain/<stubdom-id>/device-model/<target-id>/state (command specific value).
634. Write "running" back to /local/domain/<stubdom-id>/device-model/<target-id>/state.
64
65Defined commands:
66 - "pci-ins" - PCI hot plug, results:
67   - "pci-inserted" - success
68   - "pci-insert-failed" - failure
69 - "pci-rem" - PCI hot remove, results:
70   - "pci-removed" - success
71   - ??
72 - "save" - save domain state to console 1, results:
73   - "paused" - success
74 - "continue" - resume domain execution, after loading state from console 2 (require -loadvm command argument), results:
75   - "running" - success
76
77
78Toolstack to Linux ioemu stubdomain protocol
79--------------------------------------------
80
81This section describe communication protocol between toolstack and
82qemu-upstream running in Linux stubdomain. The protocol include
83expectations of both stubdomain, and qemu.
84
85Setup (done by toolstack, expected by stubdomain):
86 - Block devices for target domain are connected as PV disks to stubdomain,
87   according to configuration order, starting with xvda
88 - Network devices for target domain are connected as PV nics to stubdomain,
89   according to configuration order, starting with 0
90 - [not implemented] if graphics output is expected, VFB and VKB devices are set for stubdomain
91   (its backend is responsible for exposing them using appropriate protocol
92   like VNC or Spice)
93 - other target domain's devices are not connected at this point to stubdomain
94   (may be hot-plugged later)
95 - QEMU command line is stored in
96   /vm/<target-uuid>/image/dm-argv xenstore dir, each argument as separate key
97   in form /vm/<target-uuid>/image/dm-argv/NNN, where NNN is 0-padded argument
98   number
99 - target domain id is stored in /local/domain/<stubdom-id>/target xenstore path
100?? - bios type is stored in /local/domain/<target-id>/hvmloader/bios
101 - stubdomain's console 0 is connected to qemu log file
102 - stubdomain's console 1 is connected to qemu save file (for saving state)
103 - stubdomain's console 2 is connected to qemu save file (for restoring state)
104 - next consoles are connected according to target guest's serial console configuration
105
106Environment exposed by stubdomain to qemu (needed to construct appropriate qemu command line and later interact with qmp):
107 - target domain's disks are available as /dev/xvd[a-z]
108 - console 2 (incoming domain state) must be connected to an FD and the command
109   line argument $STUBDOM_RESTORE_INCOMING_ARG must be replaced with fd:$FD to
110   form "-incoming fd:$FD"
111 - console 1 (saving domain state) is added over QMP to qemu as "fdset-id 1" (done by stubdomain, toolstack doesn't need to care about it)
112 - nics are connected to relevant stubdomain PV vifs when available (qemu -netdev should specify ifname= explicitly)
113
114Startup:
1151. toolstack starts PV stubdomain with stubdom-linux-kernel kernel and stubdom-linux-initrd initrd
1162. stubdomain initialize relevant devices
1173. stubdomain starts qemu with requested command line, plus few stubdomain specific ones - including local qmp access options
1184. stubdomain starts vchan server on /local/domain/<stubdom-id>/device-model/<target-id>/qmp-vchan, exposing qmp socket to the toolstack
1195. qemu signal readiness by writing "running" to /local/domain/<stubdom-id>/device-model/<target-id>/state xenstore path
1206. now device model is considered running
121
122QEMU can be controlled using QMP over vchan at /local/domain/<stubdom-id>/device-model/<target-id>/qmp-vchan. Only one simultaneous connection is supported and toolstack needs to ensure that.
123
124Limitations:
125 - PCI passthrough require permissive mode
126 - only one nic is supported
127 - at most 26 emulated disks are supported (more are still available as PV disks)
128 - graphics output (VNC/SDL/Spice) not supported
129
130
131                                   PV-GRUB
132                                   =======
133
134  This replaces pygrub to boot domU images safely: it runs the regular grub
135inside the created domain itself and uses regular domU facilities to read the
136disk / fetch files from network etc. ; it eventually loads the PV kernel and
137chain-boots it.
138
139Configuration
140=============
141
142In your PV config,
143
144- use pv-grub.gz as kernel:
145
146kernel = "pv-grub.gz"
147
148- set the path to menu.lst, as seen from the domU, in extra:
149
150extra = "(hd0,0)/boot/grub/menu.lst"
151
152or you can provide the content of a menu.lst stored in dom0 by passing it as a
153ramdisk:
154
155ramdisk = "/boot/domU-1-menu.lst"
156
157or you can also use a tftp path (dhcp will be automatically performed):
158
159extra = "(nd)/somepath/menu.lst"
160
161or you can set it in option 150 of your dhcp server and leave extra and ramdisk
162empty (dhcp will be automatically performed)
163
164Limitations
165===========
166
167- You can not boot a 64bit kernel with a 32bit-compiled PV-GRUB and vice-versa.
168To cross-compile a 32bit PV-GRUB,
169
170export XEN_TARGET_ARCH=x86_32
171
172- bootsplash is supported, but the ioemu backend does not yet support restart
173for use by the booted kernel.
174
175- PV-GRUB doesn't support virtualized partitions. For instance:
176
177disk = [ 'phy:hda7,hda7,w' ]
178
179will be seen by PV-GRUB as (hd0), not (hd0,6), since GRUB will not see any
180partition table.
181
182
183                                Your own stubdom
184                                ================
185
186  By running
187
188cd stubdom/
189make c-stubdom
190
191  or
192
193cd stubdom/
194make caml-stubdom
195
196  you can compile examples of C or caml stub domain kernels.  You can use these
197and the relevant Makefile rules as basis to build your own stub domain kernel.
198Available libraries are libc, libxc, libxs, zlib and libpci.
199