CentOS – Using NAT with KVM Guests

Wednesday, July 17th, 2019

CentOS – Using NAT with KVM Guests

Please note that all commands in this guide must be run on the main HOST machine (the physical machine).  They should not be run on KVM guests (virtual machines).

If your server has a limited number of IPv4 addresses, it might be best to setup and run some virtual machines that are configured to use the default NAT network interface that KVM provides.  This will allow you to run multiple virtual machines that share the same IP address.  Think of it as setting up a home network with multiple devices with certain ports forwarded to specific devices for incoming connections.

First, create the virtual machines that are going to use NAT using virt-manager as you normally would.  In the virtual machine configuration wizard, assign the default NAT network interface named "virbr0".  After the virtual machines have been created, shut down the virtual machines.  Now, we'll assign these virtual machines static LAN IP addresses so that we can port forward certain ports and always have them reach the proper virtual machines. 

The first thing we need to do is get the MAC address of each virtual machine.  Write down the name of the virtual machine and its MAC address, as we'll need this information later on when we edit the NAT interface and assign static LAN IP addresses to our virtual machines.  Run this command to retrieve the MAC address for a specific virtual machine.

virsh dumpxml VM_NAME | grep -i '<mac'

Using the MAC address information from the VMs we want to use NAT with, edit the default NAT interface by running the below command. 

virsh net-edit default

If for some reason the NAT interface is not named default, you can find it by running the below command:

virsh net-list

After the <range /> entry, assign the static LAN IP addresses similar to the following:

<dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:ff:4a:2a' name='vm1' ip='192.168.122.13'/>
      <host mac='52:54:00:bb:35:67' name='vm2' ip='192.168.122.14'/>
      <host mac='52:54:00:aa:d9:f2' name='vm3' ip='192.168.122.15'/>
</dhcp>

Save the file with your desired values and quit the editor.  Restart the NAT interface by running the below commands:

virsh net-destroy default
virsh net-start default

Now, you'll need to setup your iptables port forwarding rules.  Adjust the below rules as necessary (changing the port numbers to the ones you want to use) and then save them so that they persist:

iptables -I FORWARD -o virbr0 -d 192.168.122.13 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39989 -j DNAT --to 192.168.122.13:39989
iptables -I FORWARD -o virbr0 -d 192.168.122.14 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39990 -j DNAT --to 192.168.122.14:39990
iptables -I FORWARD -o virbr0 -d 192.168.122.15 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39991 -j DNAT --to 192.168.122.15:39991
iptables -t nat -A POSTROUTING -s 192.168.122.0/24 -j MASQUERADE
iptables -A FORWARD -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i virbr0 -o br0 -j ACCEPT
iptables -A FORWARD -i virbr0 -o lo -j ACCEPT
iptables-save
service iptables save

Congrats, your virtual machines are now using NAT, have been assigned static LAN IP addresses, and iptables rules on the host server have been configured to port forward specific ports to each NAT VM.


Persistently Saving NAT Port Forward Rules

The only solution I found that would persistently save my NAT forwarding rules is to create a libvirt hook bash script as mentioned here

service iptables stop
iptables -F
service iptables save
service iptables start
mkdir -p  /etc/libvirt/hooks
nano  /etc/libvirt/hooks/qemu

The contents of the "/etc/libvirt/hooks/qemu" file should look similar to the following:

#!/bin/bash
# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "vm1" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.13
   GUEST_PORT=39989
   HOST_PORT=39989   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm2" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.14
   GUEST_PORT=39990
   HOST_PORT=39990   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm3" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.15
   GUEST_PORT=39991
   HOST_PORT=39991
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

Save and exit.  Make the script executable.

chmod +x /etc/libvirt/hooks/qemu

Reboot the host server.


Old Instructions for Persistent Saving (Non-Working)

If your iptables forwarding rules are not persisted after the host machine is rebooted or shutdown, run the following commands:

sudo -i
yum install -y iptables-services
systemctl stop firewalld
systemctl disable firewalld
systemctl enable iptables
nano /etc/sysconfig/iptables-config 

Change the below values to "yes":

IPTABLES_SAVE_ON_RESTART="yes"
IPTABLES_SAVE_ON_STOP="yes"

Save and exit.  Reboot the server.

If you're still having issues, try this (will clear your existing iptables rules):

iptables-save > iptables_bk
service iptables stop
iptables -F
<run iptables NAT rules here>
<run any other iptables rules you want>
service iptables save
service iptables start

More Detailed Guide

Copying LVM Containers from One Remote Server to Another

Saturday, April 27th, 2019

Transferring LVM Containers

Before you transfer a KVM container to another machine, create a KVM virtual machine on the target server with the same or larger disk size than the container being transferred. 

You can see a full list of LVM containers by using the below command:

sudo lvdisplay

Copying an LVM Container from the Local Machine to a Remote Server

sudo -i
dd if=/dev/vms/phpdev bs=4096 | pv | ssh root@IPADDRESS_HERE -p SSH_PORT 'dd of=/dev/pool/phpdev bs=4096'

Adjust the above pool paths as necessary since this may vary from server to server. 

Copying an LVM Container from a Remote Machine to the Local Machine

sudo -i
ssh root@IPADDRESS_HERE -p SSH_PORT "dd if=/dev/vms/phpdev bs=4096" | dd of="/dev/vms/phpdev" bs="4096"

Adjust the above pool paths as necessary since this may vary from server to server. 

With SSH Passphrase Key

If you're using an SSH key that is protected with a passphrase, use the below commands to open the key, provide the passphrase for that key, and copy the containers without being prompted for the passphrase when the container transfer begins:

sudo -i
eval $(ssh-agent)
ssh-add /root/keys/{PATH_TO_KEY}
dd if=/dev/pool/test bs=4096 | pv | ssh root@host.com -p {PORT} -i /root/keys/{PATH_TO_KEY} 'dd of=/dev/haha/test bs=4096'

CentOS 7 – Easiest Way to Configure LVM KVM Pool for Virtual Machines

Saturday, April 27th, 2019

Configuring LVM in CentOS

When installing CentOS 7, be sure to only partition the hard drive with about 100GB of space for the OS file system itself.  Leave the rest of the drive unpartitioned.  After CentOS has been successfully installed, run gparted via a terminal using the below command:

sudo gparted

Create a new "LVM2 PV" file system based partition on the drive's remaining space like so:

Now, create the LVM volume group by using the below command and replacing /dev/md126p3 with the new partition's path label:

sudo vgcreate vms /dev/md126p3

Now, launch virt-manager by running the below command:

sudo virt-manager

Go to "Edit" –> "Connection Details" –> click on the "Storage" tab.  Click on the "+" icon on the bottom left.  You're now creating a storage pool.  Give it a name like "vms" which is short for virtual machines.  Select "logical: LVM Volume Group" for the type.  Here's a screenshot:

In "Target Path" select the volume group that you created named vms (which you did earlier using the "vgcreate" command).  Do NOT check the "Build Pool" checkbox, and leave the "Source Path" field blank.  Here's a screenshot of what it should look like:

Click on "Finish".  You're done, and you can now create LVM storage containers for your KVM configured pool named vms.

Here's a good LVM KVM Pool guide from RedHat that includes more information (though it's not as simple as following this guide).