The first step is to copy /images/xen/vmlinuz and /images/xen/initrd.img from the Fedora (or RHEL or CentOS) DVD somewhere convenient, I use /boot/OS/ (where OS is the name of the image) but other locations will do.
Now choose a suitable Ethernet MAC address for the interface (see my previous post on how I choose them [1]).
Create a temporary block device for the install, I use /dev/VG0/OS-install (where OS is replaced by the name of the distribution, “f8” or “cent5“), it’s a logical volume in an LVM volume group named VG0. The device should be at least 2G in size for a basic Fedora install (512M for swap, 1G for files, and 512M free after the install). It is of course possible to use DOS partitions for the Xen block devices, but this would be unreasonably difficult to manage. An option for people who don’t like LVM would be to use files on an XFS filesystem (Ext3 performs poorly when creating and removing large files).
When configuring Xen on Debian systems I generally use /dev/hda type device names. The device name seems quite arbitrary and /dev/hda is a familiar name for hard drives that many people have been used to for 15+ years. But the Fedora install process doesn’t like it and I’m forced to use /dev/xvda etc.
I often install Fedora on a machine that only has 256M of RAM spare for the DomU. For recent versions of Fedora 256M of RAM is about the minimum for an install at the best of times, and a HTTP install takes even more because the root filesystem used for the install is copied via HTTP and stored in a RAM disk. It might be possible to use less RAM with a CD or DVD install or even a NFS install, but I couldn’t get CD/DVD installation working and I generally don’t give Xen DomU’s NFS access if I can avoid it. So I had to create a swap space (an attempt to do an install with 256M of RAM and no swap aborted when installing the kernel package). I expect that most serious use of Xen will have 256M of RAM or less for the DomU, part of the problem here is that Xen allocates RAM not virtual memory. VMWare allocates virtual memory so the total memory for virtual machines can be greater than physical RAM and thus this problem will be less common with VMWare.
I believe that the best way of configuring virtual machine images is to have the virtual machine manager (Xen in this case) provide block devices to the virtual machine and have the virtual machine implement no partitioning (no LVM or anything equivalent). The main reason is that DOS partition tables and LVM configuration on a block device used by Xen can not be used easily in the host environment (the Dom0 for Xen). I am not aware of how to access DOS partition tables (although I’m sure it’s possible somehow) and while LVM can be used it’s a bad idea due to the fact that there is no way to deactivate a LVM volume group that is active, and the fact that there is no support for having multiple volume groups of the same name. The lack of support for multiple volume groups of the same name is a reasonable limitation, but an insurmountable problem when using a virtual machine environment. It’s quite reasonable to create several cloned instances of a virtual machine and renaming an LVM volume group would require more changes inside the virtual machine than you would want. Also using snap-shots of old versions of the virtual machine data is difficult if the same volume group name is used.
So for ease of management I want to have filesystems on block devices (such as /dev/xvda) instead of partitions (such as /dev/xvda1). Unfortunately Anaconda (the Fedora installer) doesn’t support this. So I had to do the initial install with DOS partitions and then fix it afterwards. So use the manual option and create a primary partition for the root filesystem and then create a non-primary partition for swap (when using small amounts of RAM such as 256M) so that swap can be used during the install. The root filesystem needs to be at the start of the disk to make it easier to sort this out later.
After installing Fedora and shutting the virtual machine down the next step is to copy the block device to the desired configuration (filesystem on an unpartitioned device). If the root filesystem is the first partition then the first 63 sectors will be the partition table and reserved space so dd can be used to copy the data with the following commands:
dd if=/dev/VG0/OS-install of=/dev/VG0/OS bs=512 skip=63
e2fsck -f /dev/VG0/OS
resize2fs /dev/VG0/OS
The next step is to mount the device /dev/VG0/OS in the Dom0 to change /etc/fstab, I use /dev/xvda for the root device and /dev/xvdb for swap.
Now to remove the cruft:
Avahi is a network service discovery system, mainly used for laptops and isn’t needed on a server, it is installed by default on all recent Fedora, RHEL and CentOS releases but it is not useful in a DomU (any unused network service is a security risk if you don’t disable or remove it). Smartmontools is for detecting impending failure of a hard drive and does not do any good when using a virtual block device (you run it on the Dom0). It might be considered a bug that smartd doesn’t exit on startup when it sees a device such as /dev/xvda. The pcsc-lite package for managing smart cards is of no use to me and all the other people who don’t own readers for smart-cards, and it can therefore be removed. Bluetooth networking support (in the package bluez-utils) is also only usable in a Dom0 (AFAIK), and the only bluetooth device I own is my mobile phone so I can’t use it on my computer. The command “yum remove avahi smartmontools pcsc-lite bluez-utils” removes them.
For almost all of my DomU’s I don’t use NFS or do any printing, so I remove the packages related to them. Also autofs is in most cases only useful for servers when mounting NFS filesystems. I remove them with the command “yum remove nfs-utils portmap cups autofs“.
The GPM daemon (which supports cut/paste operations with a mouse on virtual consoles) is of no use on a Xen DomU, unfortunately the vim-enhanced package depends on it. I could just disable the daemon, but as I like to run small images I remove it with “yum remove gpm“. I may have to reinstall it on some images as some of my clients like the extra VIM functionality.
It’s unfortunate that debootstrap doesn’t work on CentOS (and presumably Fedora) so installing a Debian DomU on a CentOS/Fedora Dom0 requires creating an image on a Debian machine or downloading an image from www.jailtime.org .
Sample Xen Config for the install:
kernel = “/boot/OS/vmlinuz”
ramdisk = “/boot/OS/initrd.img”
memory = 256
name = “OS”
vif = [ ‘mac=00:16:3e:66:66:68, bridge=xenbr0’ ]
disk = [ ‘phy:/dev/VG0/OS-install,xvda,w’ ]
extra=”askmethod text”
Sample Xen Config for operation:
kernel = “/boot/cent5/vmlinuz-2.6.18-53.el5xen”
ramdisk = “/boot/cent5/initrd-2.6.18-53.el5xen.img”
memory = 256
name = “cent5”
vif = [ ‘mac=00:16:3e:66:66:68, bridge=xenbr0’ ]
disk = [ ‘phy:/dev/VG0/cent5,xvda,w’, ‘phy:/dev/VG0/cent5-swap,xvdb,w’ ]
root = “/dev/xvda ro”
“It’s unfortunate that debootstrap doesn’t work on CentOS”
xen-tools says to works with CentOS (with rinse ?)
Don’t you like xen-tools?
Awesome! Thanks.
I have been trying to figure this out for a while (though not had much time). I will certainly be trying this soon.
There is an ‘rpmstrap’ package out there (inspired by debbootstrap, and packaged for Debian, too). It doesn’t look like it’s maintained anymore, but the last time I poked at it, it was easy enough to get working for more recent releases.
Also a few minor tips. . . kpartx is a useful tool for mapping partitions on a device (where device can be a loopback mounted filesystem-in-a-file/disk image, or a partition/LVM that has been sub-partitioned as a as a real disk). Once they’re mapped with kpartx, they can be accessed and mounted directly.
Also, when you say you want the host to provide raw block devices to the Xen guest, to make things easier, I think you’re making things harder and more complicated for yourself than you need to.
I frequently build Xen guests by creating two LVM partitions/volumes from the dom0 host: guest.root, guest.swap. I next use mkfs/mkswap to directly format those volumes, then I rsync an existing install (even running installs will generally work just fine) directly to the now mounted /dev/vg/guest.root filesystem. After that, I run through the mounted guest filesystem to fix up anything that needs updating (IP address, hostname, etc. . . sometimes I just boot to single user mode and fix it there). Finally, I unmount the filsystem and tell Xen to use /dev/vg/guest.root as /dev/sda1 (or /dev/xda1, or whatever). It then boots, and works, easy as that.
Now, admittedly, this only allows you to create a Xen guest as a clone of an existing install. However, it’s easy enough to do a separate “normal” install to use for cloning. And, once you’ve done it once, you can save off the contents of the root filesystem as a tarball and creating a new instance is as easy as formatting a partition and copying out the tarball.
One other thing that I’ve found really useful, is pygrub. It’s a Xen bootloader that you can use instead of specifying a kernel that is external to the Xen guest. It was written by someone at Red Hat, and is included with their Xen packages. I’ve never tried using it on Debian, but I can’t imagine it would be difficult, and it makes some things (like upgrades) easier. pygrub will come up when starting a xen guest, read the guest’s /boot, find it’s grub menu list, present it, then let you choose your kernel (which must exist on the guest’s filesystem).
i5513: I prefer to manage things myself and the past descriptions of xen-tools have not enticed me to try them out. From your comment I decided to check it out, it seems that xen-tools will use rpmstrap to install RPM based distributions. It only supports Fedora up to version 8 and CentOS 4, but I expect I could extend it to work with newer versions if necessary (Fedora 8 is a sym-link to Fedora 6 so obviously in some cases no patches are required).
However my comment about debootstrap not working on CentOS is in regard to using a CentOS Dom0 and a Debian DomU.
Christopher: Thanks for the kpartx tip!
You say that you think I’m making it harder for myself by using “raw block devices”. However what I do is very close to what you describe, I provide multiple LVM volumes and then use them as /dev/xvda, /dev/xvdb, etc. Using kpartx to get access to the data is certainly easier than my method.
http://nvd.nist.gov/nvd.cfm?cvename=CVE-2007-4993
I didn’t like pygrub before the above came out, I like it less now.
You are correct, the way we use installed guest’s are very similar. I probably should have been more specific in that I think the way you installed the guest was perhaps more complicated than it needed to be. By trying to install it with the forced partition setup that you did, and then changing the guest in place, it adds complexity. I’ve found that cloning an install is a lot easier (even if it’s a non-xen install).
I can understand your concern with pygrub, especially related to the security issue there. I guess it didn’t concern me that much because in all places where I use Xen, the same people have root access to the dom0’s and the domU’s, making it a non-issue.
The biggest thing I like about pygrub is the ability to have a fully self-contained guest image, including the kernel it is booting with. It also allows me to upgrade the guest kernel within the guest.