KVM-QEMU-Libvirt NAT and Firewall Networking
It's been a while since I've started using KVM/QEMU in conjunction with Archipel. I can say that this has been ANYTHING BUT an easy journey. My previous findings… that while the core virtualization product might be ready for prime time but the management tools are not… is proving to be an understatement. I keep running into missing features and functionality, flakey errors, and a variety of situations which mean downtime for basic changes and updates. Not to mention several situations where I've had to simply reboot the server itself to correct odd behavior and bugs.
Given the time sink that fighting through various problems presents, I will probably be migrating my development infrastructure back to VMware products before too long. This problem has taken me 8.5 hours of continuous work to resolve today on top of the 10+ hours of previous research and troubleshooting. However, I thought I'd post this since it took a lot of time to figure out.
To summarize, I ran into a pretty serious problem that I've been fighting for a while. Whenever you set up your VM environment with the intent to utilize NAT and /24 netmasks (255.255.255.0) you'll run into some challenges.
I'm not going to explain this process but in order to make this work, your router will need to have routes added to each of the different networks you create, if you're using a Cisco SOHO class router, this is the ONLY way you're going to be able to do it effectively as 255.255.0.0 is not an option. Really this isn't such a bad thing as it's probably the better way to set it up anyhow, but it took the better part of 2 months of on and off work and trying to use different subnetting before finding a solid solution.
The next problem you'll run into is that once you get up and running, while your Hypervisor will probably be accessible across them, access to your virtual machines (guests) will NOT work between the different subnets even after you've put routing in place. This is because the firewall entries that libvirt creates tend to restrict communication to the local LAN segment. While this is a great concept in theory, there's no good documentation or place where this is clearly explained. As a result, you'll probably spend hours of time digging through Google searches looking for answers before finding anything useful.
The concept is much like a router; or for even better example, home router that you're trying to configure to host a website out of your home. You have to open ports on your routers firewall to allow that to happen. The case is exactly the same here.
So, in order to do this, the easiest way I was able to find was to utilize the qemu hook for libvirt. I found this as a result of looking at these sites and you should also review them for additional background:
This explains a lot of the setup.
This script is not clear and failed to work on my debian system.
This script worked like a charm but fails to allow for multiple hosts and multiple ports.
Since the research that I did couldn't yield anything that met what I was looking for, I worked through my own solution as I'm detailing now. It is NOT perfect by any means, but it functions for my needs and hopefully can help you out.
I decided I needed to have a list of hosts and port information that would be read into this. That way I could simply add one line of information with 4 variables each time I needed an entry. It does involve some duplication of information which could be avoided with more code but I was looking for a workable solution, not to spend time scripting.
NOTE: Before we start, I recommend you have the VM you're looking to grant access to powered down. You don't necessarily have to, but given all the other quirks I've run into I feel it's safer.
First, you'll need to add yourself a file (server_port_map) in your /etc/libvirt/hooks/ directory (create the hooks directory if you haven't already, it's not something that comes with the install).
mkdir /etc/libvirt/hooks cd /etc/libvirt/hooks touch server_port_map
Now you can open the file in your favorite text editor and add an entry. The format for entries in this file is as follows:
- Each port you want to forward must be a new entry
- You can use the same vm and ip over and over again
- Each line must contain four comma separated values
- The values are: <guest name>,<ip address>,<source port>,<destination port>
- An array is used in the programming so the entries MUST be in this order
Once you have this done, you'll next need to create the qemu file in the same location.
Change it to allow execution
chmod +x qemu
Now open qemu in your favorite text editor and drop in the following code:
#!/bin/bash ### ### Port mapping script to allow access to guest virtual machine ### services from systems outside of the native subnet. ### ### Version: 2.1 ### Date: 03/09/13 ### ### Huge thanks to irc.freenode.net #bash (zendeavor/geirha) for ### their help with resolving problems due to my bash shortfalls. ### ### # Specify system and file variables ### datafile="/etc/libvirt/hooks/server_port_map" iptables='/sbin/iptables' ### # Reads in contents of file server_port_map into the HOSTARRAY array. ### while IFS=, read -r guestname guestip hostpt guestpt ; do ### # Takes the values of the variables above and creates the iptables # entries as specified. ### if [ $1 = $guestname ] ; then if [[ $2 == @(stopped|reconnect) ]] ; then $iptables -t nat -D PREROUTING -p tcp --dport $hostpt -j DNAT \ --to $guestip:$guestpt $iptables -D FORWARD -d $guestip/32 -p tcp -m state --state NEW,RELATED,ESTABLISHED \ -m tcp --dport $guestpt -j ACCEPT $iptables -t nat -D OUTPUT -p tcp -o lo --dport $hostpt -j DNAT \ --to $ip:$dpt fi if [[ $2 == @(start|reconnect) ]] ; then $iptables -t nat -I PREROUTING -p tcp --dport $hostpt -j DNAT \ --to $guestip:$guestpt $iptables -I FORWARD -d $guestip/32 -p tcp -m state --state NEW,RELATED,ESTABLISHED \ -m tcp --dport $guestpt -j ACCEPT $iptables -t nat -I OUTPUT -p tcp -o lo --dport $hostpt -j DNAT \ --to $guestip:$guestpt fi fi done < $datafile
Save your new shiny qemu file and next you should restart services, specifically the libvirt daemon; on debian that's as follows:
service libvirt-bin restart
This should cycle it and load your new firewall rules. Once complete you can start up your VM's and see if you can access them. If you're looking for a quick test you can do that from console:
virsh start <guest>
and when you're done
virsh shutdown <guest>
If you're running a 3rd party management utility like I am, you will probably want to launch the VM from there once you've completed this since it may not otherwise show up as running.
Paco Colomer · April 8, 2013 at 7:38 pm
I was selfmotivated on test Archipel and even found the solution cool, but reading your comment it looks that the technology is not mature enough for production environment, isn't it?
Due your experiencia, would you recommend for production usage? I just found this technology looking for SolusVM alternatives, and at first glance seems great idea the XMPP communication between nodes.
tcrass · April 18, 2013 at 1:32 pm
Thanks for inquiring.
I have spent countless hours on and off over the course of several months attempting to bring the Archipel/KVM solution to a point where I could use it in a live hosting environment with stability and managability. Basically I was looking for an alternative to VMWare's ESXi solutions.
Like you I found the solution quite promising in the long run, but unfortunately it's proven buggy and not currently enterprise ready. You might look at my older article HERE which I go into my installation and analysis of the technology.
Because of the problems that I've run into, at this point I've actually had to abandon this environment completely and I'm working on setting up a VMWare ESXi environment again in its place.
I may explore this option again in the future but for now, this is NOT a viable solution until futher development is completed.
Paco Coloemr · April 24, 2013 at 4:14 am
Thanks your answer, very helpful.
I agree, looks like a promising product but unmature for production environtments. I will keep an eye every now and then to see how the project developes but will not take any risk implementing it.