Linux KVM:  Using Multiple NICs and Routing Traffic Properly Between Them

When setting up a KVM guest to use multiple network interface controllers (NICs), additional ip routes may be needed in order for the additional interfaces to work properly.  For example, if you configure a NIC with a public static IP address and a NIC with an internal private DHCP LAN IP address, you must create a route for any traffic that comes through the DHCP LAN IP address to respond via the interface from which the request originated.  Otherwise, forwarded NAT traffic from the main KVM host to the DHCP internal LAN IP will reach its destination, but no response will be sent back (because it will attempt to send the response via the configured static IP address interface which may NOT be the original destination of the senders request).

The Solution:

https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming/23345#answer-23345

From the above link, the solution for me was to do the following in the KVM guest virtual machine:

Only needs to be done once:

sudo -i
echo 200 isp1 >> /etc/iproute2/rt_tables

Setting up the route (adjust variables as necessary):

sudo -i
ip rule add from <interface_IP> table isp1 priority 900
ip rule add from <interface_IP> dev <interface> table isp1
ip route add default via <gateway_IP> dev <interface> table isp

The command I used for my specific setup:

sudo -i
ip rule add from 192.168.122.10 table isp1 priority 900 
ip rule add from 192.168.122.10 dev ens9 table isp1 
ip route add default via 192.168.122.1 dev ens9 table isp1

Making it permanent (apply on system start up):

sudo -i
nano /etc/network/interfaces

I added the below post-up rules (adjust variables as necessary):

auto ens9
iface ens9 inet dhcp
        post-up ip rule add from <interface_IP> table isp1 priority 900
        post-up ip rule add from <interface_IP> dev <interface> table isp1
        post-up ip route add default via <gateway_IP> dev <interface> table isp1

The route is created whenever the dhcp interface is brought up.

Obtaining Let's Encrypt HTTP Validation Server IP Addresses

Use your webserver logs:

sudo apt-get install john
cat access_log.1 | grep "Let's Encrypt" | awk '{print $1}' | unique ips
cat ips

Installing Chrome WebDriver (Linux Script)

Find out which version of Chrome is installed on your system before running the below commands.  You can find out your chrome version by running the following command:

google-chrome --version

Adjust the version number (replace {VERSION_NUMBER})  in the below commands to match the version installed on your system!!!

sudo -i
cd ~/Downloads
rm chromedriver_linux64.zip
wget -N https://chromedriver.storage.googleapis.com/{VERSION_NUMBER}/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver

Selenium and other libraries that rely on the Chrome WebDriver should now work properly.

Installing the Newest Version of Python 2.7.x on Older Ubuntu Systems

If you need to upgrade to the newest version of Python 2.7.x, and you're running an older distribution (like Ubuntu 14.04), use the following commands to get and compile the latest version from source (works on Ubuntu 17.04 and older – tested on Ubuntu 14.04):

sudo apt-get install build-essential checkinstall
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
version=2.7.18
cd ~/Downloads/
wget https://www.python.org/ftp/python/$version/Python-$version.tgz
tar -xvf Python-$version.tgz
cd Python-$version
./configure --with-ensurepip=install
make
sudo make install

Install requests and hashlib:

sudo rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
sudo rm -r /usr/lib/python2.7/dist-packages/chardet
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
sudo pip install requests
sudo easy_install hashlib

You may need to create a symlink for chardet after installing it directly from pip:

ln -sf /usr/local/lib/python2.7/site-packages/chardet /usr/lib/python2.7/dist-packages/chardet

If you get the error of "ImportError: cannot import name _remove_dead_weakref" when running a pam python based authentication script after installing the new version of python, try this fix:

sudo cp /usr/local/lib/python2.7/weakref.py /usr/local/lib/python2.7/weakref_old.py
sudo cp /usr/lib/python2.7/weakref.py /usr/local/lib/python2.7/weakref.py

Getting Let's Encrypt Certbot to Work:

Now, you'll need to delete the EFF directory from the /opt directory to avoid old configuration issues that were used for your older version of python.  Once you cleanup this directory, you'll run certbot again so it can reconfigure itself. 

sudo rm -r /opt/eff.org/
sudo certbot

Old Way

jonathonf is now a very greedy person and has made his repositories private, so this method no longer works as of 12/20/2019.

sudo add-apt-repository ppa:jonathonf/python-2.7
sudo apt-get update
sudo apt-get install python2.7

Then, you'll need to cleanup a few leftover system packages manually before installing the newest version of python-pip.  If you don't do this, you'll run into problems installing some new packages using pip.

sudo rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
sudo rm -r /usr/lib/python2.7/dist-packages/chardet
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so

Now, you can download and install the newest version of python-pip:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py

Getting Let's Encrypt Certbot to Work:

First, you'll need to install a few packages that Certbot (the Let's Encrypt client) uses:

sudo pip install requests
sudo pip install hmac

Now, you'll need to delete the EFF directory from the /opt directory to avoid old configuration issues that were used for your older version of python.  Once you cleanup this directory, you'll run certbot again so it can reconfigure itself. 

sudo rm -r /opt/eff.org/
sudo certbot

You're done.

Full list of commands (for quickly doing all of the above):

sudo -i
add-apt-repository ppa:jonathonf/python-2.7
apt-get update
apt-get install python2.7
rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
rm -r /usr/lib/python2.7/dist-packages/chardet
rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
mkdir -p /root/Downloads
cd /root/Downloads
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install requests
pip install hmac
rm -r /opt/eff.org/
certbot

Use nano as the Default Editor

If you hate vi like I do, you can configure Linux to always default to using the nano editor.

Simply add the following to the bottom of the /etc/bashrc file:

export EDITOR="nano"

Save the file.  nano is now the default editor.  When you use

sudo crontab -e

The nano editor will now be used by default.

Transferring LVM Containers

Before you transfer a KVM container to another machine, create a KVM virtual machine on the target server with the same or larger disk size than the container being transferred. 

You can see a full list of LVM containers by using the below command:

sudo lvdisplay

Copying an LVM Container from the Local Machine to a Remote Server

sudo -i
dd if=/dev/vms/phpdev bs=4096 | pv | ssh root@IPADDRESS_HERE -p SSH_PORT 'dd of=/dev/pool/phpdev bs=4096'

Adjust the above pool paths as necessary since this may vary from server to server. 

Copying an LVM Container from a Remote Machine to the Local Machine

sudo -i
ssh root@IPADDRESS_HERE -p SSH_PORT "dd if=/dev/vms/phpdev bs=4096" | dd of="/dev/vms/phpdev" bs="4096"

Adjust the above pool paths as necessary since this may vary from server to server. 

With SSH Passphrase Key

If you're using an SSH key that is protected with a passphrase, use the below commands to open the key, provide the passphrase for that key, and copy the containers without being prompted for the passphrase when the container transfer begins:

sudo -i
eval $(ssh-agent)
ssh-add /root/keys/{PATH_TO_KEY}
dd if=/dev/pool/test bs=4096 | pv | ssh root@host.com -p {PORT} -i /root/keys/{PATH_TO_KEY} 'dd of=/dev/haha/test bs=4096'

PolicyKit pkexec – Running without Prompting for Authentication

The following guide explains how to configure a pkexec command to run without prompting for authentication.  This is helpful when you want to grant root access to key piece of the system (such as allowing virsh commands from another user when running KVM virtual machines) or just want to run a GUI command as root without having to login or use authentication. 

https://askubuntu.com/questions/383747/how-to-configure-pkexec-to-not-ask-for-password#answer-388660

Or in our own archive in case the above link disappears.

Location of Ubuntu Startup Application Scripts

All Versions of Ubuntu

In all versions of Ubuntu, startup scripts can be configured and run from the following locations:

/etc/init/*.conf – some init scripts
/etc/rc.local – a file that is run by root on system boot (bash scritps and other commands can go in here)
~/.config/autostart – user specific GUI programs that are run once the X11 environment is started
/etc/xdg/autostart – Global GUI programs that are run once the X11 environment is started
@reboot cronjob – cronjob scripts that are executed when the system boots

Ubuntu 16.04 and Later

systemd init scripts in /etc/systemd/system/*.service files
systemd init scripts in /lib/systemd/system/*.service files

Network Manager Applet NOT Showing in MATE Desktop

If you install the MATE desktop environment on Ubuntu, your Network Manager icon may not show up in the taskbar as shown below:

Without the Network Manager icon showing, you will have a hard time managing and connecting to WIFI networks.  To get it to show up, you will need to edit the following file using nano:

sudo nano /etc/xdg/autostart/nm-applet.desktop

Look for the below line:

AutostartCondition=GNOME3 unless-session gnome

And comment it out like so:

#AutostartCondition=GNOME3 unless-session gnome

Save the file using "Ctrl + O", and then exit nano using "Ctrl + X".  Restart your computer.  The network icon will show up again as shown below:

Note, the actual icon will vary based on the selected MATE desktop theme.  The screenshots above show the icon used in the LUNA theme.

Save iptables Rules on Shutdown, Restore on Boot

When your server shutsdown (halts), reboots, or enters runlevel 1, your iptables configuration is automatically wiped and reset.  Packages such as iptables-persistent supposedly help with this problem, but unless you save your rules manually, the rules are never saved automatically when the system reboots. 

I've wanted to truly persist my iptables, so I decided to change their package to always save the iptables rules when the system reboots, halts, or enters runlevel 1 automatically.  The rules are then restored when the system boots to runlevels 2-5.  This means that your iptables configuration will persist forever.  This may not be desired, but if I ban an IP address permanently, I always want it to be banned.

If you'd like to use this modified version of iptables-persistent so that your rules are automatically saved on shutdown, you can install it by running the below commands:

sudo apt-get remove iptables-persistent
sudo dpkg -r iptables-persistent
wget http://dinofly.com/files/linux/iptables-persistent_0.5.8_all.deb
sudo dpkg --install iptables-persistent_0.5.8_all.deb

Use at your own risk.  If you do something stupid, it will persist until you can clear it!

The above package was tested in Ubuntu 12.04 x86, Ubuntu 12.04 x64, Ubuntu 14.04 x86, Ubuntu 14.04 x64, and Ubuntu 15.04 x64.  Should work on other debian operating systems as well.