README for the Xen packages
===========================

This file contains SUSE-specific instructions and suggestions for using Xen.

For more in-depth documentation of using Xen on SUSE, consult the
virtualization chapter in the SLES or SUSE Linux manual, or read up-to-date
virtualization information, including a list of known issues, at
http://www.novell.com/documentation/vmserver/.

For more complete documentation on Xen itself, please install one of the
xen-doc-* packages and read the documentation installed into
/usr/share/doc/packages/xen/.


About
-----
Xen allows you to run multiple virtual machines on a single physical machine.

See the Xen homepage for more information:
  http://www.cl.cam.ac.uk/research/srg/netos/xen/

If you want to use Xen, you need to install the Xen hypervisor and a number of
supporting packages.  During the initial SUSE installation (or when installing
from YaST) check-mark the "Xen Virtual Machine Host Server" pattern.  If,
instead, you wish to install Xen manually later, click on the "Install
Hypervisor and Tools" icon in YaST.

If you want to install and manage VMs graphically, be sure to install a 
graphical desktop environment like KDE or GNOME.  The following optional 
packages are needed to manage VMs graphically. Note that "Install Hypervisor 
and Tools" installs all the packages below:
  vm-install       (Optional, to install VMs)
  virt-manager     (Optional, to manage VMs graphically)
  virt-viewer      (Optional, to view VMs outside virt-manager)

Additional packages:
  nbd-client       (Optional, to access virtual disks stored on NBD servers)
  open-iscsi       (Optional, to access virtual disks stored on iSCSI targets)

You then need to reboot your machine.  Instead of booting a normal Linux
kernel, you will boot the Xen hypervisor and a slightly changed Linux kernel.
This Linux kernel runs in the first virtual machine and will drive most of
your hardware.

This approach is called paravirtualization, since it is a partial
virtualization (the Linux kernel needs to be changed slightly, to make the
virtualization easier).  It results in very good performance (consult
http://www.cl.cam.ac.uk/research/srg/netos/xen/performance.html) but has the
downside of unchanged operating systems not being supported.  However, new
hardware features (e.g., Intel VT and AMD-V) are overcoming this limitation.


Terminology
-----------
The Xen open-source community has a number of terms that you should be
familiar with.

A "domain" is Xen's term for a virtual machine.

"Domain 0" is the first virtual machine.  It can control all other virtual
machines.  It also (usually) controls the physical hardware.  A kernel used in
domain 0 may sometimes be referred to as a dom0 kernel.

"Domain U" is any virtual machine other than domain 0.  The "U" indicates it
is unprivileged (that is, it cannot control other domains).  A kernel used in
an unprivileged domain may be referred to as a domU kernel.

Novell documentation will use the more industry-standard term "virtual
machine", or "VM", rather than "domain" where possible.  And to that end,
domain 0 will be called the "virtual machine server", since it essentially the
server on which the other VMs run.  All other domains are simply "virtual
machines".

The acronym "HVM" refers to a hardware-assisted virtual machine.  These are
VMs that have not been modified (e.g., Windows) and therefore need hardware
support such as Intel VT or AMD-V to run on Xen.


Kernels
-------
Xen supports two kinds of kernels: A privileged kernel (which boots the
machine, controls other VMs, and usually controls all your physical hardware)
and unprivileged kernels (which can't control other VMs, and usually don't need
drivers for physical hardware).  The privileged kernel boots first (as the VM
server); an unprivileged kernel is used in all subsequent VMs.

The VM server takes control of the boot process after Xen has initialized the
CPU and the memory.  This VM contains a privileged kernel and all the hardware
drivers.

For the other virtual machines, you usually don't need the hardware drivers.
(It is possible to hide a PCI device from the VM server and re-assign it to
another VM for direct access, but that is a more advanced topic.)  Instead you
use virtual network and block device drivers in the unprivileged VMs to access
the physical network and block drivers in the VM server.

For simplicity, SUSE ships a single Xen-enabled Linux kernel, rather than
separate privileged and unprivileged kernels.  As most of the hardware drivers
are modules anyway, using this kernel as an unprivileged kernel has very
little extra overhead.

The kernel is contained in the kernel-xen package, which you need to install to
use Xen.


Booting
-------
If you installed Xen during the initial SUSE installation, or installed one
of the kernel-xen* packages later, a "XEN" option should exist in your Grub
bootloader.  Select that to boot SUSE on top of Xen.

If you want to add additional entries, or modify the existing ones, you will
have to edit Grub yourself.  All Xen entries in the Grub configuration file
(usually /boot/grub/menu.lst) look something like this:

title XEN
    root (hd0,5)
    kernel /xen.gz
    module /vmlinuz-xen <parameters>
    module /initrd-xen

Replace (hd0,5) with the partition that holds your /boot directory in
grub-speak, e.g., hda1 -> (hd0,0) and sda5 -> (hd2,4).

Normally, xen.gz requires no parameters.  If you want to add parameters,
see below.

Replace "<parameters>" with the kernel parameters that you want to pass to
your kernel.  These should be very similar, if not identical, to those passed
to a normal kernel that you boot on bare iron.

Once you have booted this configuration successfully, you are running Xen with
a privileged kernel on top of it.


Xen Boot Parameters
-------------------
Normally, xen.gz requires no parameters.  However, in special cases (such as
debugging or a dedicated VM server) you may wish to pass it parameters.

In particular in case of problems you may want to attach a serial terminal and
direct Xen to send its output not only to the screen, but also to that
terminal. In order to do so, add "console=vga,com<n> com<n>=<baud>" (without
the quotes and with <n> replaced by the serial port number - generally 1 or 2 -
and with <baud> replaced by the baud rate the serial terminal is configured
for) to the xen.gz line.

For a more complete discussion of possible parameters, see the user
documentation in the xen-doc-* packages.


Init scripts
------------
Before you can create additional VMs (or use any other xm command) xend must
be running.  This init script is part of the xen-tools package, and it is
activated at installation time.  You can (de)activate it using insserv.  You
can also start it manually with "rcxend start".

The deprecated xendomains script is also shipped, but disabled by default.  In
SLES 10 GA (xen 3.0.2) and older, this script allowed VMs to be started and
stopped automatically when the machine starts and stops.  In SLES 10 SP1 (xen
3.0.4) and newer, the proper way to start and stop VMs automatically is to set
the "on_xend_start" and "on_xend_stop" settings in the VMs configuration.
(Deprecating xendomains was necessary because xend, not the configuration file
in /etc/xen/vm, is now the authoritative source for the VM's settings.)
Consult the online documentation for more information.


Creating a VM with vm-install
-----------------------------
The vm-install program (part of the vm-install package, and accessible
through YaST's Control Center) is the recommended method to create VMs.  This
program handles creating both the VM's configuration file and disk(s).  It can
help install any operating system, not just SUSE.

From the command line, run "vm-install".  If the DISPLAY environment variable
is set and the supporting packages (python-gtk) are installed, a graphical
wizard will start.  Otherwise, a text wizard will start.

Each VM needs to have its own root filesystem.  The root filesystem can live
on a block device (e.g., a hard disk partition, or an LVM2 or EVMS volume) or
in a file that holds the filesystem image.

VMs can share filesystems, such as /usr or /opt, that are mounted read-only
from _all_ VMs.  Never try to share a filesystem that is mounted read-write;
filesystem corruption will result.  For sharing writable data between VMs, use
NFS or other networked or cluster filesystems.

When defining the virtual network adapter(s), we recommend using a static MAC
for the VM rather than allowing Xen to randomly select one each time the VM
boots.  (See "Network Troubleshooting" below.)  XenSource has been allocated a
range of MAC addresses with the OUI of 00-16-3E.  By using MACs from this
range you can be sure they will not conflict with any physical adapters.

Once you have the VM configured, click "OK".  The wizard will now create a
configuration file for the VM, and create a disk image.  The disk image will
exist in /var/lib/xen/images, and a corresponding config file will exist in
/etc/xen/vm.  The operating system's installation program will then run within
the VM.

When the VM shuts down (because the installation -- or at least the first
stage of it -- is done), the wizard finalizes the VM's configuration and
restarts the VM.

The creation of VMs can be automated; read the vm-install man page for more
details.  The installation of an OS within the VM can be automated if the OS
supports it.


Creating a VM Manually
----------------------
If you create a VM manually (as opposed to using vm-install, which is the
recommended way), you will need to create a disk (or reuse an existing one)
and a configuration file.

If you are using a disk or disk image that is already installed with an
operating system, you'll probably need to replace its kernel with a
Xen-enabled kernel.

The kernel and ramdisk used to bootstrap the VM must match any kernel modules
that might be present in the VM's disk.  It is possible to manually copy the
kernel and ramdisk from the VM's disk (for example, after updating the kernel
within that VM) to the VM server's filesystem.  However, an easier (and less
error-prone) method is to use something called the "domUloader".  Before a new
VM is started, this loader automatically copies the kernel and ramdisk into
the VM server's filesystem, so that it can be used to bootstrap the new VM.
See /etc/xen/examples/xmexample.domUloader for an example.

Next, make a copy of one of the /etc/xen/examples/* files, and modify it to
suit your needs.  For paravirtualized VMs, start with
/etc/xen/examples/xmexample1; for fully virtualized VMs, start with
/etc/xen/examples/xmexample.hvm.  You'll need to change (at very least) the
"name" and "disk" parameters.


Managing Virtual Machines
-------------------------
VMs can be managed from the command line or from virt-manager.

Before a VM can be started, xend must be informed of it.  vm-install will
automatically import new VM configurations into xend.  However, if you copy a
VM from another machine, or manually create a VM configuration file, you will
need to import it into xend with a command like:
  xm new my-vm
If your VM's configuration file is not located in /etc/xen/vm, you must
specify the full path.  This imports the configuration into xend (and
therefore virt-manager becomes aware of it, also).

Now to start the VM:
  xm start my-vm
or start it graphically from virt-manager.

Have a look at running sessions with "xm list".  Note the ID of the newly
created VM.  Attach to the VM's text console with "xm console <ID>" (replacing
ID with the VM's ID).  Attaching to multiple VM consoles is most conveniently
done with the terminal multiplexer "screen".

Have a look at the other xm commands by typing "xm help".  Note that most xm
commands must be done as root.


Using the Mouse via VNC in Fully Virtual Mode
---------------------------------------------
In a fully virtualized VM, the mouse may be emulated as a PS/2 mouse, USB
mouse, or USB tablet.  The vm-install tool selects the best emulation that is
known to be automatically detected and supported by the operating system.

However, when accessing some fully virtualized operating systems via VNC, the
mouse may be difficult to control if the VM is emulating a PS/2 mouse.  PS/2
provides mouse deltas, but VNC only provides absolute coordinates.  In such
cases, you may want to manually switch the operating system and VM to use a
USB tablet.

Emulation of a SummaSketch graphics tablet is provided for this reason.  To
use the Summa emulation, you will need to configure your fully virtualized OS.
Note that the virtual tablet is connected to the second virtual serial port
(/dev/ttyS1 or COM2).

Most Linux distributions ship with appropriate drivers, and only need to be
configured.  To configure gpm, edit /etc/sysconfig/mouse and add these lines:
MOUSETYPE="summa"
XMOUSETYPE="SUMMA"
DEVICE=/dev/ttyS1
The format and location of your configuration file could vary depending upon
your Linux distribution.  The goal is to run the gpm daemon as follows:
  gpm -t summa -m /dev/ttyS1
X also needs to be configured to use the Summa emulation.  Add the following
stanza to /etc/X11/xorg.conf, or use your distribution's tools to add these
settings:
Section "InputDevice"
	Identifier "Mouse0"
	Driver "summa"
	Option "Device" "/dev/ttyS1"
	Option "InputFashion" "Tablet"
	Option "Mode" "Absolute"
	Option "Name" "EasyPen"
	Option "Compatible" "True"
	Option "Protocol" "Auto"
	Option "SendCoreEvents" "on"
	Option "Vendor" "GENIUS"
EndSection
After making these changes, restart gpm and X.

To ensure the VM is emulating a USB tablet, add these lines to the
configuration file in /etc/xen/vm:
  usb=1
  usbdevice='tablet'
Then re-import the configuration into xend:
  xm new my-vm


HVM Console in Fully Virtual Mode
---------------------------------
When running a VM in fully virtual mode, a special console is available that
provides some additional ways to control the VM.  Press Ctrl-Alt-2 to access
the console; press Ctrl-Alt-1 to return to the VM.  While at the console,
type "help" for help.

The two most important commands are "send-key" and "change".  The "send-key"
command allows you to send any key sequence to the VM, which might otherwise
be intercepted by your local window manager.

The "change" command allows the target of a block device to be changed; for
example, use it to change from one CD ISO to another.  Some versions of Xen
have this command disabled for security reasons.  Consult the online
documentation for workarounds.


Networking
----------
Your virtual machines become much more useful if you can reach them via the
network.  Starting with openSUSE11.1 and SLE11, networking in domain 0 is
configured and managed via YaST.  The yast2-networking module can be used
to create and manage bridged networks.  During initial installation, a bridged
networking proposal will be presented if the "Xen Virtual Machine Host Server"
pattern is selected.  The proposal will also be presented if you install Xen
after initial installation using the "Install Hypervisor and Tools" module in
YaST.

The default proposal creates a virtual bridge in domain 0 for each active
ethernet device, enslaving the device to the bridge.  Consider a machine
containing two ethernet devices (eth0 and eth1), both with active carriers.
YaST will create br0 and br1, enslaving the eth0 and eth1 devices repectively.

VMs get a virtual network interface (e.g. eth0), which is visible in domain 0
as vifN.0 and connected to the bridge.  This means that if you set up an IP
address in the VMs belonging to the same subnet as br0 from your domain 0,
you'll be able to communicate not only with the other slave VMs, but also with
domain 0 and with the external network.  If you have a DHCP server running in
your network, your VMs should succeed in getting an IP address.

Be aware that this may have unwanted security implications.  You may want to
opt for routing instead of bridging, so you can set up firewalling rules in
domain 0.

Please read about the network configuration in the Xen manual.  You can set up
bridging or routing for other interfaces also.

For debugging, here's what happens on bootup of a domU:
- xenstored saves the device setup in xenstore
- domU is created
- vifN.0 shows up in domain 0 and a hotplug event is triggered
- hotplug is /sbin/udev; udev looks at /etc/udev/rules.d/40-xen.rules and
  calls /etc/xen/scripts/vif-bridge online
- vif-bridge set the vifN.0 device up and enslaves it to the bridge
- eth0 shows up in domU (hotplug event triggered)
Similar things happen for block devices, except that /etc/xen/scripts/block is
called.

It's not recommended to use ifplugd nor NetworkManager for managing the
interfaces if you use bridging mode.  Use routing with nat or proxy-arp
in that case.  You also need to do that in case you want to send out packets
on wireless; you can't bridge Xen "ethernet" packets into 802.11 packets.


Thread-Local Storage
--------------------
For some time now, the glibc thread library (NPTL) has used a shortcut to
access thread-local variables at a negative segment offset from the segment
selector GS instead of reading the linear address from the TDB (offset 0).
Unfortunately, this optimization has been made the default by the glibc and
gcc maintainers, as it saves one indirection.  For Xen this is bad: The access
to these variables will trap, and Xen will need to use some tricks to make the
access work.  It does work, but it's very slow.

SUSE Linux 9.1 and SLES 9 were prior to this change, and thus are not
affected.  SUSE Linux 9.2 and 9.3 are affected.  For SUSE Linux 10.x and SLES
10, we have disabled negative segment references in gcc and glibc, and so
these are not affected.  Other non-SUSE Linux distributions may be affected.

For affected distributions, one way to work around the problem is to rename
the /lib/tls directory, so the pre-i686 version gets used, where no such
tricks are done.  An example LSB-compliant init script which automates these
steps is installed at /usr/share/doc/packages/xen/boot.xen.  This script
renames /lib/tls when running on Xen, and restores it when not running on Xen.
Modify this script to work with your specific distribution.

Mono has a similar problem, but this has been fixed in SUSE Linux 10.1 and
SLES 10.  Older or non-SUSE versions of Mono may have a performance impact.


Security
--------
Domain 0 has control over all domains. This means that care should be taken to
keep domain 0 safe; ideally you strip it down to only do as little there as
possible, preferably with no local users except for the system administrator.
Most commands in domain 0 can only be performed as root, but this protection
scheme only has moderate security and might be defeated.  In case domain 0 is
compromised, all other domains are compromised as well.

To allow relocation of VMs (migration), the receiving machine listens on TCP
port 8002.  You might want to put firewall rules in place in domain 0 to
restrict this to machines which you trust. You have some access control in
xend-config.sxp as well by tweaking the xend-relocation-hosts-allow
setting.  Relocating VMs with sensitive data is not a good idea in untrusted
networks, since the data is not sent encrypted.

The memory protections for the domUs are effective; so far no way to break out
of a virtual machine is known.  A VM is an effective jail.


Limitations
-----------
When booting, Linux reserves data structures matching the amount of RAM found.
This has the side-effect that you can't dynamically grow the memory beyond
what the kernel has been booted with.  But you can trick domU Linux to prepare
for a larger amount of RAM by passing the mem= boot parameter.

The export of virtual hard disks from files in Xen can be handled via the
loopback driver (although in Xen >= 3.0.4, this is can be replaced by the
"blktap" user-space driver.)  If you are still using loopback, it may be
possible to run out of loopback devices, as by default only 64 are supported.
You can change this by inserting:
options loop max_loop=128
into /etc/modprobe.conf.local in domain 0.


Network Troubleshooting
-----------------------
First ensure the VM server is configured correctly and can access the network.

Do not use ifplugd or NetworkManager, neither are bridge aware.

Specify a static virtual MAC in the VM's configuration file.  Random MACs can
be problematic, since with each boot of the VM it appears that some hardware
has been removed (the previous random MAC) and new hardware is present (the
new random MAC).  This can cause network configuration files (which were
intended for the old MAC) to not be matched up with the new virtual hardware.

In the VM's filesystem, ensure the ifcfg-eth* files are named appropriately.
For example, if you do decide to use a randomly-selected MAC for the VM, the
ifcfg-eth* file must not include the MAC in its name; name it generically
("ifcfg-eth0") instead.  If you use a static virtual MAC for the VM, be sure
that is reflected in the file's name.


Troubleshooting
---------------
First try to get Linux running on bare iron before trying with Xen.

Be sure your Xen hypervisor (xen) and VM kernels (kernel-xen) are compatible.
The hypervisor and domain 0 kernel are a matched set, and usually must be
upgraded together.  Consult the online documentation for a matrix of supported
32- and 64-bit combinations

On certain machines with 2GB or less of RAM, domain 0 Linux may fail to boot,
printing the following messages:
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
...
Kernel panic - not syncing: PCI-DMA: Memory would be corrupted
Fix this by adding "swiotlb=16" to the Linux kernel command line, which
reserves additional memory for the swiotlb (the actual number to be used here
of course depends on the system configuration).

If you have trouble early in the boot, try passing pnpacpi=off to the Linux
kernel.  If you have trouble with interrupts or timers, passing lapic to Xen
may help.  Xen and Linux understand similar ACPI boot parameters.  Try the
options acpi=off,force,strict,ht,noirq or acpi_skip_timer_override.

Other useful debugging options to Xen may be nosmp, noreboot, mem=1024M,
sync_console, noirqbalance (Dell).  For a complete list of Xen boot options,
consult chapter 11.3 of the Xen users' manual.

If domain 0 Linux crashes on X11 startup, please try to boot into runlevel 3.

To debug Xen or domain 0 Linux crashes or hangs, it may be useful to use the
debug-enabled hypervisor, and/or to prevent automatic rebooting.  Change your
Grub configuration from something like this:
    kernel (hd0,5)/xen.gz
To something like this:
    kernel (hd0,5)/xen-dbg.gz noreboot
After rebooting, the Xen hypervisor will write any error messages to the log
file (viewable with the "xm dmesg" command).

If problems persist, check if a newer version is available.  Well-tested
versions will be shipped with SUSE and via YaST Online Update.  More frequent
(but less supported) updates are available on Novell's Forge site:
  http://forge.novell.com/modules/xfmod/project/?xenpreview


Upgrading the Host Operating System
-----------------------------------
When upgrading the host operating system from one major release to another 
(for example, SLES 10 to SLES 11 or openSUSE 11.4 to openSUSE 12.1) or when
applying a service pack like SLES 11 SP2 to SLES 11 SP1 all running VMs must
be shut down before the upgrade process is begun.


Memory Ballooning in VMs
------------------------
Setting a VMs maximum memory value greater than the initial memory value
requires support for memory ballooning in the VMs operating system. Modern SLES
and openSUSE guests have this capabilitity built-in. Windows installation media
does not support memory ballooning so you must first install the VM without
memory ballooning (maxmem equal to initial memory). After the installation, the
Virtual Machine Driver Pack (vmdp) must be installed. After this, the VMs
maxmem value may be increased. A reboot of the VM is required for this action
to take effect.


Known Issues
------------
For a list of known issues and work-arounds, see
http://www.novell.com/documentation/vmserver/.


Disclaimer
----------
Xen performed amazingly well in our tests and proved very stable.  Still, you
should be careful when using it, just like you'd be careful if you boot an
experimental kernel.  Expect that it may not boot and be prepared to have a
fall-back solution for that scenario.  Be prepared that it may not support all
of your hardware.  And for the worst of all cases, have your most valuable
data backed up.  (This is always a good idea, of course.)


Feedback
--------
In case you have remarks about, problems with, ideas for, or praise for Xen,
please report it back to the xen-devel list:
  xen-devel@lists.xensource.com
If you find issues with the packaging or setup done by Novell/SUSE, please
report it to:
  http://www.suse.de/feedback/


				ENJOY!
			Your Novell SUSE Team.