Category Archives: Networking

2

Xen and Bridging

|

In a default configuration of Xen there will be a virtual Ethernet device created for each interface which will be associated with a bridge. A previous post documented how to configure a bridge named xenbr0.

The basic configuration of Xen that most people use is to have a single virtual Ethernet port for each Xen instance and have them all connected to the one bridge, and then the Dom0 will have an IP address on the bridge interface that is used for routing packets to the outside world. This works really well if you have a subnet that you are using for all Xen DomU IP addresses, if you are using NAT for communication, or if the DomU needs no communication outside the Dom0 and other DomU’s on the same machine (a common case for testing).

But if you have a collection of servers that you want to consolidate on a single piece of hardware then you end up using a single sub-net that spans some physical machines, some Xen Dom0’s, and some DomU’s. The solution to this is to use bridged networking.

Unfortunately most documentation of bridged networking is really confusing, and non of my google searches turned up the most relevant fact:

When setting up a bridge on the local Ethernet you must make your physical ethernet device (eth0 or whatever) be strictly a slave to the bridge and then assign the IP address used for the physical network to the bridge.

ifconfig eth0 up
brctl addif xenbr0 eth0

For example if you have 10.0.0.42 being the IP address used by the Dom0 on the local Ethernet via device eth0 and you want to use bridging for DomU’s then you simply make eth0 owned by xenbr0 (the typical name for the Xen bridge) with the above commands in your script to configure the xenbr0 device. Then treat xenbr0 in the same way that you treated eth0 before enabling bridging.

Also there’s nothing stopping you from having one bridge for DomU’s that can talk directly to the physical Ethernet and another for DomU’s that are only to use routed networking, see my previous post about using multiple ethernet devices in Xen for more background information.

Related posts:

  1. multiple ethernet devices in Xen It seems that no-one has documented what needs to be...
  2. installing Xen domU on Debian Etch I have just been installing a Xen domU on Debian...
  3. Ethernet bonding Bonding is one of the terms used to describe multiple...
  4. Xen and eth device renaming Recently I rebooted one of my Debian Xen servers and...
  5. first look at CentOS 5 Xen I have just installed a machine running CentOS 5 as...

Ethernet bonding

|

Bonding is one of the terms used to describe multiple Ethernet cables used to form a single virtual network link. This can be done for performance or reliability.

Bonding for performance used to be common when 100baseT was the fastest network technology that was commonly available. In 1999 servers could usually sustain considerably more than 10MB/s so a single 100baseT network interface was a performance bottleneck. At that time I worked with Cisco switches and Solaris machines that had up to four 100baseT links bonded for performance.

Nowadays Gigabit Ethernet is commonly available, most laptops have Gigabit Ethernet on the motherboard. Gigabit PCI cards are as cheap as $35, and Gigabit switches can be purchased for as little as $139. Server hardware is a little more expensive, but it’s still quite cheap and commonly available.

Most people don’t need more than Gigabit speed, in fact most systems can not saturate a Gigabit link due to poor application design, a slow operating system, or slow disks used to provide the data. So at this time there is little speed for bonded Gigabit networking for performance.

There is still the issue of reliability. Often you want to have two ethernet cards and cables configured so that if one breaks the network won’t go down.

One annoying thing about bonding in Linux (in 2.6.x kernels) is that the module has to be loaded separately for each bond interface, and the parameters for an interface can’t be changed without unloading and loading the driver (very painful if you log in to the machine via ssh over the bonded interface to do sys-admin work).

The parameters I have in /etc/modprobe.conf for bonding are:

alias bond0 bonding
options bond0 mode=1 arp_interval=500 arp_ip_target=192.168.0.1

This means that if there is no traffic on the link then every 500ms an ARP request will be sent for the address 127.128.129.130 (I used the address of my router but substituted a different value for this blog entry). An ARP request for a machine on the local LAN is a request that will always be satisfied if the machine in question and the network link are working.

The idea is that you have two switches and every computer that matters has two ethernet ports. If one port stops working (broken Ethernet card, cable, or router) then the other takes over.

The special file /proc/net/bonding/bond0 can be used to view the current configuration of the bond0 device.

Below are sample configuration files for Fedora and Red Hat Enterprise Linux to configure bonding:

/etc/sysconfig/networking/devices/ifcfg-bond0:
DEVICE=bond0
IPADDR=192.168.0.2
NETMASK=255.255.255.0
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
ONBOOT=yes
BOOTPROTO=static
# GATEWAY should be the IP address to ARP ping
GATEWAY=192.168.0.1
TYPE=Ethernet

/etc/sysconfig/networking/devices/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
SLAVE=yes
MASTER=bond0
TYPE=Ethernet

/etc/sysconfig/networking/devices/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
SLAVE=yes
MASTER=bond0
TYPE=Ethernet

Note that there is nothing preventing you from having more than two devices bonded together for reliability, but I doubt that you really need that.

No related posts.