Wednesday, June 19, 2013

Accessing services on KVM guests behind a NAT

I have a small web service running on a Fedora 17 VM. The VM lives on the default virtual network provided by libvirt, which allows outbound connections to the external world. But because the IPv4 space for that network (192.168.122.0/24) is private, the host's iptables rules will NAT all packets on their way out.

That means "no way in" for external connections. The virtual network addresses are unseen (unroutable) to the outside world.

Luckily, libvirtd provides hooks so that you can insert custom operations at key points in the start-up/shut-down phases of a guest.   The following procedure shows how to create a hook to modify iptables when a guest is starting (or stopping).
  1. On the host, create a new file called /etc/libvirt/hooks/qemu containing this sample script. (There's a link to a more comprehensive script if you need it. I also wrote a Python replacement below.)
  2. Replace the placeholder variables in the sample script with the guest name, addresses, and ports that are to be forwarded.
  3. Restart the libvirt service.
  4. Verify new DNAT rules with iptables -nvL -t nat.
Make sure that your guest doesn't have any iptables rules that will interfere.  For instance, you'll need to open up TCP port 80 if you want your web service to be accessible.

As an exercise, I rewrote the sample script in Python, adding some logging and removing some redundancy along the way.

#!/usr/bin/env python
#!/usr/bin/env python
import argparse
import logging
import subprocess
import sys

guestAddr = { 'f17-base':'192.168.122.200' }
hostPort = { '80':'8000', '22':'222' }
logFile = '/var/log/qemu-hook.log'
logLevel = logging.DEBUG

logging.basicConfig(filename=logFile, level=logLevel, format='%(asctime)s : %(message)s')

parser = argparse.ArgumentParser()
for arg in [ 'guest', 'op', 'subop', 'extra' ]:
    parser.add_argument(arg)
args = parser.parse_args()

logging.debug('qemu hook: guest %s, op %s' % (args.guest, args.op))
if args.guest not in guestAddr:
    logging.debug('Nothing to do for guest %s' % args.guest)
    sys.exit()

if args.op == 'start':
    natOp = '-A'
    filterOp = '-I'
    opStr = 'Adding'
elif args.op == 'stopped':
    natOp = filerOp = '-D'
    opStr = 'Removing'
else:
    logging.debug('Nothing to do for op %s' % args.op)
    sys.exit()

for gport in hostPort.keys():
    # build the nat command
    natcmd = ['iptables', '-t', 'nat', natOp, 'PREROUTING']
    gAddrPort = "{}:{}".format(guestAddr[args.guest], gport)
    natcmd += ['-p', 'tcp', '--dport', hostPort[gport], '-j', 'DNAT', '--to', gAddrPort]  
    logging.info('%s nat rules for %s' % (opStr, gAddrPort))
    subprocess.call(natcmd)
    # build the filter command
    filtercmd = ['iptables', filterOp, 'FORWARD']
    filtercmd += ['-d', '{}/32'.format(guestAddr[args.guest]), '-p', 'tcp']
    filtercmd += ['-m', 'state', '--state', 'NEW']
    filtercmd += ['-m', 'tcp', '--dport', gport, '-j', 'ACCEPT']
    logging.info('%s forwarding rules for %s' % (opStr, gport))
    subprocess.call(filtercmd)

logging.info('iptables update complete')
sys.exit()

Note that, depending on the version of libvirtd you are running, the hook may not process "reconnect" operations -- i.e., when libvirtd is restarted.  You need 0.9.13 or later for that.  If you have an earlier version, you may have an issue with duplicate rules.

I left those operations out of the Python script because my Fedora 17 system is running 0.9.11.

Cheers!

Fixed DHCP addresses in libvirt

The libvirt DHCP service can deal out fixed IP addresses.  Yes, you could simply configure the guest OS with a static IP, but that's beside the point.  A DHCP service should be able to handle fixed addresses and, earlier in the game, I didn't think libvirt was capable of it.  But it is!

Just add a host element to the default network XML file using virsh net-edit default.

<network>
  <name>default</name>
  <uuid>5c0448c7-4240-4325-9a80-eb9f575c962e</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:BE:58:83'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.128' end='192.168.122.254' />
      <host mac='52:54:00:c9:06:c5' name='router122' ip='192.168.122.200' />
    </dhcp>
  </ip>
</network>

Of course, this same principle applies to other virtual networks.  The reason I doubted it at first is that virt-manager 0.9.5 doesn't expose this level of customization in its GUI.

Everything you need to know about network configuration can be found at libvirt networking.