Archive for the ‘Linux’ Category

The Dangers of Using tcp_tw_recycle in Linux – Strange Intermittent Timeout Issues

Wednesday, January 13th, 2021

Do Not Use tcp_tw_recycle

I had a very strange connectivity issue recently that I was only able to reproduce intermittently on my own LAN network when connecting to a few of my servers hosting websites that process and receive tons of simultaneous connections at any point in time.  Basically, my connection to a specific set of websites that I host would timeout from my home internet connection.  However, I was never able to reproduce this issue when connecting to the same sites from other networks belonging to my family and friends. 

From my home connection, I used TCPView and saw that SYN_SENT packets were supposedly sent to my servers to establish a connection.  Unfortunately, the server never replied to some of these requests.  As such, my connection would timeout at times, and work perfectly fine sometimes.  I looked at DD-WRT's connection table, and it also claimed that the packets had been sent, but that they were in an UNREPLIED state when I experienced issues.  Thus, packets were supposedly being sent, but the server was not responding at times.  After spending nearly a week trying to tackle this issue and buying new cable internet equipment (an officially supported Comcast modem), I tracked down the issue, and it ended up being a TCP configuration setting on my servers rather than my home LAN equipment.

Modem or Router's Fault?

Originally, I thought my issue was caused by the DD-WRT open source firmware I was running on my wireless router.  If I restored the router's settings to DD-WRT's factory defaults, I could always connect to the websites I was having intermittent connection timeout issues on.  I suspected it might be my router after trying an older router which didn't have any problems either.  I even upgraded the DD-WRT firmware to the latest version and rebuilt my complicated network configuration from scratch.  Unfortunately, the issue was still there.  Thus, despite mixed results with different routers, I started to wonder if the issue was on my server's end.

Finally Fixed

I started looking at sysctl TCP settings I could adjust on my router, and I ended up comparing some of these values to the ones used on my servers (that were hosting the problem websites).  Eventually, I came across configuration values I had changed myself several months ago which were supposed to help the server support multiple simultaneous connections.

After reading this StackOverflow thread (https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both), I decided I would try disabling the tcp_tw_recycle setting.

/proc/sys/net/ipv4/tcp_tw_recycle was set to 1 (enabled) from tweaks I had run that I had found on the internet.  After I disabled it, /proc/sys/net/ipv4/tcp_tw_recycle was set to 0 (disabled).  By default, Linux keeps tcp_tw_recycle disabled.  Again, this is something I had changed for tuning reasons.  After disabling this setting and rebooting the server, I no longer have any issues connecting to the severs in question.  No more connection timeouts, and everything works properly again.

I have no idea why I wasn't able to reproduce this issue on other networks.  I thought it was my network equipment (modem and router), but it ended up being the server.

Lessons Learned

Be careful when applying settings you find online.  Sometimes, they may not work, or their usage may be buggy.  In fact, net.ipv4.tcp_tw_recycle has been removed from Linux in kernel versions newer than 4.12 by default.  I'm guessing this is because it doesn't work, as I experienced.  Do NOT use  net.ipv4.tcp_tw_recycle! I kept tcp_tw_reuse enabled, so you can enable this setting without running into problems.  Just don't for the love of anything use tcp_tw_recycle!  It doesn't work, and it will cause you headaches trying to track down strange intermittent issues!

 

Linux Multiple Network Interfaces (NICs) – One Interface with Static Public IP and One Interface with Private DHCP LAN IP Address – Routes and Routing

Friday, July 24th, 2020

Linux KVM:  Using Multiple NICs and Routing Traffic Properly Between Them

When setting up a KVM guest to use multiple network interface controllers (NICs), additional ip routes may be needed in order for the additional interfaces to work properly.  For example, if you configure a NIC with a public static IP address and a NIC with an internal private DHCP LAN IP address, you must create a route for any traffic that comes through the DHCP LAN IP address to respond via the interface from which the request originated.  Otherwise, forwarded NAT traffic from the main KVM host to the DHCP internal LAN IP will reach its destination, but no response will be sent back (because it will attempt to send the response via the configured static IP address interface which may NOT be the original destination of the senders request).

The Solution:

https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming/23345#answer-23345

From the above link, the solution for me was to do the following in the KVM guest virtual machine:

Only needs to be done once:

sudo -i
echo 200 isp1 >> /etc/iproute2/rt_tables

Setting up the route (adjust variables as necessary):

sudo -i
ip rule add from <interface_IP> table isp1 priority 900
ip rule add from <interface_IP> dev <interface> table isp1
ip route add default via <gateway_IP> dev <interface> table isp

The command I used for my specific setup:

sudo -i
ip rule add from 192.168.122.10 table isp1 priority 900 
ip rule add from 192.168.122.10 dev ens9 table isp1 
ip route add default via 192.168.122.1 dev ens9 table isp1

Making it permanent (apply on system start up):

sudo -i
nano /etc/network/interfaces

I added the below post-up rules (adjust variables as necessary):

auto ens9
iface ens9 inet dhcp
        post-up ip rule add from <interface_IP> table isp1 priority 900
        post-up ip rule add from <interface_IP> dev <interface> table isp1
        post-up ip route add default via <gateway_IP> dev <interface> table isp1

The route is created whenever the dhcp interface is brought up.

Installing Chrome WebDriver (Linux Script)

Wednesday, August 28th, 2019

Installing Chrome WebDriver (Linux Script)

Find out which version of Chrome is installed on your system before running the below commands.  You can find out your chrome version by running the following command:

google-chrome --version

Adjust the version number (replace {VERSION_NUMBER})  in the below commands to match the version installed on your system!!!

sudo -i
cd ~/Downloads
rm chromedriver_linux64.zip
wget -N https://chromedriver.storage.googleapis.com/{VERSION_NUMBER}/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver

Selenium and other libraries that rely on the Chrome WebDriver should now work properly.

CentOS – Using NAT with KVM Guests

Wednesday, July 17th, 2019

CentOS – Using NAT with KVM Guests

Please note that all commands in this guide must be run on the main HOST machine (the physical machine).  They should not be run on KVM guests (virtual machines).

If your server has a limited number of IPv4 addresses, it might be best to setup and run some virtual machines that are configured to use the default NAT network interface that KVM provides.  This will allow you to run multiple virtual machines that share the same IP address.  Think of it as setting up a home network with multiple devices with certain ports forwarded to specific devices for incoming connections.

First, create the virtual machines that are going to use NAT using virt-manager as you normally would.  In the virtual machine configuration wizard, assign the default NAT network interface named "virbr0".  After the virtual machines have been created, shut down the virtual machines.  Now, we'll assign these virtual machines static LAN IP addresses so that we can port forward certain ports and always have them reach the proper virtual machines. 

The first thing we need to do is get the MAC address of each virtual machine.  Write down the name of the virtual machine and its MAC address, as we'll need this information later on when we edit the NAT interface and assign static LAN IP addresses to our virtual machines.  Run this command to retrieve the MAC address for a specific virtual machine.

virsh dumpxml VM_NAME | grep -i '<mac'

Using the MAC address information from the VMs we want to use NAT with, edit the default NAT interface by running the below command. 

virsh net-edit default

If for some reason the NAT interface is not named default, you can find it by running the below command:

virsh net-list

After the <range /> entry, assign the static LAN IP addresses similar to the following:

<dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:ff:4a:2a' name='vm1' ip='192.168.122.13'/>
      <host mac='52:54:00:bb:35:67' name='vm2' ip='192.168.122.14'/>
      <host mac='52:54:00:aa:d9:f2' name='vm3' ip='192.168.122.15'/>
</dhcp>

Save the file with your desired values and quit the editor.  Restart the NAT interface by running the below commands:

virsh net-destroy default
virsh net-start default

Now, you'll need to setup your iptables port forwarding rules.  Adjust the below rules as necessary (changing the port numbers to the ones you want to use) and then save them so that they persist:

iptables -I FORWARD -o virbr0 -d 192.168.122.13 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39989 -j DNAT --to 192.168.122.13:39989
iptables -I FORWARD -o virbr0 -d 192.168.122.14 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39990 -j DNAT --to 192.168.122.14:39990
iptables -I FORWARD -o virbr0 -d 192.168.122.15 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39991 -j DNAT --to 192.168.122.15:39991
iptables -t nat -A POSTROUTING -s 192.168.122.0/24 -j MASQUERADE
iptables -A FORWARD -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i virbr0 -o br0 -j ACCEPT
iptables -A FORWARD -i virbr0 -o lo -j ACCEPT
iptables-save
service iptables save

Congrats, your virtual machines are now using NAT, have been assigned static LAN IP addresses, and iptables rules on the host server have been configured to port forward specific ports to each NAT VM.


Persistently Saving NAT Port Forward Rules

The only solution I found that would persistently save my NAT forwarding rules is to create a libvirt hook bash script as mentioned here

service iptables stop
iptables -F
service iptables save
service iptables start
mkdir -p  /etc/libvirt/hooks
nano  /etc/libvirt/hooks/qemu

The contents of the "/etc/libvirt/hooks/qemu" file should look similar to the following:

#!/bin/bash
# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "vm1" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.13
   GUEST_PORT=39989
   HOST_PORT=39989   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm2" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.14
   GUEST_PORT=39990
   HOST_PORT=39990   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm3" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.15
   GUEST_PORT=39991
   HOST_PORT=39991
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

Save and exit.  Make the script executable.

Here is an example of forwarding multiple ports with one entry:
 

if [ "${1}" = "vm_name" ]; then
   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.3
   GUEST_PORTS=(15231 15232 15233)   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        for i in ${GUEST_PORTS[@]}; do
            /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
            #/sbin/iptables -t nat -D PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            /sbin/iptables -t nat -D PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -D PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -D PREROUTING -p all -j DNAT --to $GUEST_IP:${i}
        done
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        for i in ${GUEST_PORTS[@]}; do
            /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
            #/sbin/iptables -t nat -I PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            /sbin/iptables -t nat -I PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -I PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -I PREROUTING -p all -j DNAT --to $GUEST_IP:${i}
        done
   fi
fi
chmod +x /etc/libvirt/hooks/qemu

If you're attempting to forward incoming traffic (only traffic originating from the outside), be sure to exclude the main virbr0 interface so that requests originating from the DHCP LAN aren't forwarded to itself.  This is helpful for port forwarding DNS (port 53).  Incoming requests from outside of the network will be forwarded to the specified server, but outgoing DNS requests made from the server will NOT be forwarded back to itself.  Basically, you want to forward all traffic on port 53 to that specific server except for traffic that originates from that server.  So, be sure to add this line if that is the case:

/sbin/iptables -t nat -I PREROUTING \! -i virbr0 -p udp --dport 53 -j DNAT --to $GUEST_IP:$GUEST_PORT

More information can be found here:  https://serverfault.com/questions/1042383/kvm-port-forwarding-port-53-to-guest-via-nat-temporary-failure-in-name-resolut/1042394#1042394

Reboot the host server.


Old Instructions for Persistent Saving (Non-Working)

If your iptables forwarding rules are not persisted after the host machine is rebooted or shutdown, run the following commands:

sudo -i
yum install -y iptables-services
systemctl stop firewalld
systemctl disable firewalld
systemctl enable iptables
nano /etc/sysconfig/iptables-config 

Change the below values to "yes":

IPTABLES_SAVE_ON_RESTART="yes"
IPTABLES_SAVE_ON_STOP="yes"

Save and exit.  Reboot the server.

If you're still having issues, try this (will clear your existing iptables rules):

iptables-save > iptables_bk
service iptables stop
iptables -F
<run iptables NAT rules here>
<run any other iptables rules you want>
service iptables save
service iptables start

More Detailed Guide

Installing the Newest Version of Python 2.7.x on Older Versions of Ubuntu (like 14.04 and 16.04)

Thursday, May 9th, 2019

Installing the Newest Version of Python 2.7.x on Older Ubuntu Systems

If you need to upgrade to the newest version of Python 2.7.x, and you're running an older distribution (like Ubuntu 14.04), use the following commands to get and compile the latest version from source (works on Ubuntu 17.04 and older – tested on Ubuntu 14.04):

sudo apt-get install build-essential checkinstall
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
version=2.7.18
cd ~/Downloads/
wget https://www.python.org/ftp/python/$version/Python-$version.tgz
tar -xvf Python-$version.tgz
cd Python-$version
./configure --with-ensurepip=install
make
sudo make install

Install requests and hashlib:

sudo rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
sudo rm -r /usr/lib/python2.7/dist-packages/chardet
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
sudo pip install requests
sudo easy_install hashlib

You may need to create a symlink for chardet after installing it directly from pip:

ln -sf /usr/local/lib/python2.7/site-packages/chardet /usr/lib/python2.7/dist-packages/chardet

If you get the error of "ImportError: cannot import name _remove_dead_weakref" when running a pam python based authentication script after installing the new version of python, try this fix:

sudo cp /usr/local/lib/python2.7/weakref.py /usr/local/lib/python2.7/weakref_old.py
sudo cp /usr/lib/python2.7/weakref.py /usr/local/lib/python2.7/weakref.py

Getting Let's Encrypt Certbot to Work:

Now, you'll need to delete the EFF directory from the /opt directory to avoid old configuration issues that were used for your older version of python.  Once you cleanup this directory, you'll run certbot again so it can reconfigure itself. 

sudo rm -r /opt/eff.org/
sudo certbot

Old Way

jonathonf is now a very greedy person and has made his repositories private, so this method no longer works as of 12/20/2019.

sudo add-apt-repository ppa:jonathonf/python-2.7
sudo apt-get update
sudo apt-get install python2.7

Then, you'll need to cleanup a few leftover system packages manually before installing the newest version of python-pip.  If you don't do this, you'll run into problems installing some new packages using pip.

sudo rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
sudo rm -r /usr/lib/python2.7/dist-packages/chardet
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
sudo rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so

Now, you can download and install the newest version of python-pip:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py

Getting Let's Encrypt Certbot to Work:

First, you'll need to install a few packages that Certbot (the Let's Encrypt client) uses:

sudo pip install requests
sudo pip install hmac

Now, you'll need to delete the EFF directory from the /opt directory to avoid old configuration issues that were used for your older version of python.  Once you cleanup this directory, you'll run certbot again so it can reconfigure itself. 

sudo rm -r /opt/eff.org/
sudo certbot

You're done.

Full list of commands (for quickly doing all of the above):

sudo -i
add-apt-repository ppa:jonathonf/python-2.7
apt-get update
apt-get install python2.7
rm /usr/lib/python2.7/dist-packages/chardet*.egg-info
rm -r /usr/lib/python2.7/dist-packages/chardet
rm /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
rm /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
mkdir -p /root/Downloads
cd /root/Downloads
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install requests
pip install hmac
rm -r /opt/eff.org/
certbot

Change the Default Editor to nano in Linux

Saturday, April 27th, 2019

Use nano as the Default Editor

If you hate vi like I do, you can configure Linux to always default to using the nano editor.

Simply add the following to the bottom of the /etc/bashrc file:

export EDITOR="nano"

Save the file.  nano is now the default editor.  When you use

sudo crontab -e

The nano editor will now be used by default.

Configuring Let’s Encrypt Certbot on CentOS 7 with lighttpd

Saturday, April 27th, 2019

Configuring Let's Encrypt Certbot on CentOS 7 with lighttpd

Installing Certbot

First, install certbot by using the below commands:

sudo yum -y install epel-release
sudo yum install certbot

certbot is python based program that allows you to request SSL certificates for your domains. 

Request a Certificate

Use the below command to request a certificate (adjust paths and replace the test.com domain as necessary):

sudo certbot certonly --webroot -w /var/www/vhosts/test/httpdocs -d test.com

A certificate has now been stored in /etc/letsencrypt/live.  Create a combined certificate format by using the below command (replacing test.com with your real domain):

/bin/cat /etc/letsencrypt/live/test.com/cert.pem /etc/letsencrypt/live/test.com/privkey.pem > /etc/letsencrypt/live/test.com/custom.pem && /bin/chmod 777 /etc/letsencrypt/live/test.com/custom.pem && /sbin/service lighttpd restart

Certificate Renewal Cronjobs

You may want to create a cronjob to renew the certificate and a cronjob for regenerating the combined format certificate since the underlying certificate file can change (such as when it's renewed):

sudo crontab -e

Insert the below cronjobs:

0 1 * * 1 /usr/bin/certbot renew --quiet
5 1 * * 1 /bin/cat /etc/letsencrypt/live/test.com/cert.pem /etc/letsencrypt/live/test.com/privkey.pem > /etc/letsencrypt/live/test.com/custom.pem && /bin/chmod 777 /etc/letsencrypt/live/test.com/custom.pem && /sbin/service lighttpd restart

Save your crontab configuration. 

Setting Up Lighttpd to Use SSL Certificate

Edit your default-enabled lighttpd configuration file in /etc/lighttpd/vhosts.d to look similar to the following (replacing test.com with your real domain and adjusting various file paths)

$HTTP["host"] == "test.com" {
  var.server_name = "test.com"
  server.name = server_name  server.document-root = vhosts_dir + "/test/httpdocs"
  #accesslog.filename          = vhosts_dir + "/test/log" + "/access.log"
}
$SERVER["socket"] == ":80" {
  server.document-root = vhosts_dir + "/test/httpdocs"
}
$SERVER["socket"] == ":443" {
    ssl.engine           = "enable"
    ssl.pemfile          = "/etc/letsencrypt/live/test.com/custom.pem"
    server.document-root = vhosts_dir + "/test/httpdocs"
    ssl.ca-file = "/etc/letsencrypt/live/test.com/chain.pem" # Root CA
    server.name = "test.com" # Domain Name OR Virtual Host Name
}

Here's how you can set a different document root for specific https (port 443) virtual hosts:

$SERVER["socket"] == ":443" {
    ssl.engine           = "enable"
    ssl.pemfile          = "/etc/letsencrypt/live/test.com/custom.pem"
    server.document-root = vhosts_dir + "/test/httpdocs/"
    ssl.ca-file = "/etc/letsencrypt/live/test.com/chain.pem" # Root CA
    server.name = "test.com" # Domain Name OR Virtual Host Name
    
    $HTTP["host"] =~ "(^|www\.)somethingelse.test.com" {
        server.document-root = vhosts_dir + "/test/httpdocs/subdir"
    }
}

Save and restart the lighttpd service.

sudo service lighttpd restart

Congrats, SSL is now available on your domain, and your Let's Encrypt certificate has been configured and will be renewed automatically by your cronjob.

Copying LVM Containers from One Remote Server to Another

Saturday, April 27th, 2019

Transferring LVM Containers

Before you transfer a KVM container to another machine, create a KVM virtual machine on the target server with the same or larger disk size than the container being transferred. 

You can see a full list of LVM containers by using the below command:

sudo lvdisplay

Copying an LVM Container from the Local Machine to a Remote Server

sudo -i
dd if=/dev/vms/phpdev bs=4096 | pv | ssh root@IPADDRESS_HERE -p SSH_PORT 'dd of=/dev/pool/phpdev bs=4096'

Adjust the above pool paths as necessary since this may vary from server to server. 

Copying an LVM Container from a Remote Machine to the Local Machine

sudo -i
ssh root@IPADDRESS_HERE -p SSH_PORT "dd if=/dev/vms/phpdev bs=4096" | dd of="/dev/vms/phpdev" bs="4096"

Adjust the above pool paths as necessary since this may vary from server to server. 

With SSH Passphrase Key

If you're using an SSH key that is protected with a passphrase, use the below commands to open the key, provide the passphrase for that key, and copy the containers without being prompted for the passphrase when the container transfer begins:

sudo -i
eval $(ssh-agent)
ssh-add /root/keys/{PATH_TO_KEY}
dd if=/dev/pool/test bs=4096 | pv | ssh root@host.com -p {PORT} -i /root/keys/{PATH_TO_KEY} 'dd of=/dev/haha/test bs=4096'

CentOS 7 – Easiest Way to Configure LVM KVM Pool for Virtual Machines

Saturday, April 27th, 2019

Configuring LVM in CentOS

When installing CentOS 7, be sure to only partition the hard drive with about 100GB of space for the OS file system itself.  Leave the rest of the drive unpartitioned.  After CentOS has been successfully installed, run gparted via a terminal using the below command:

sudo gparted

Create a new "LVM2 PV" file system based partition on the drive's remaining space like so:

Now, create the LVM volume group by using the below command and replacing /dev/md126p3 with the new partition's path label:

sudo vgcreate vms /dev/md126p3

Now, launch virt-manager by running the below command:

sudo virt-manager

Go to "Edit" –> "Connection Details" –> click on the "Storage" tab.  Click on the "+" icon on the bottom left.  You're now creating a storage pool.  Give it a name like "vms" which is short for virtual machines.  Select "logical: LVM Volume Group" for the type.  Here's a screenshot:

In "Target Path" select the volume group that you created named vms (which you did earlier using the "vgcreate" command).  Do NOT check the "Build Pool" checkbox, and leave the "Source Path" field blank.  Here's a screenshot of what it should look like:

Click on "Finish".  You're done, and you can now create LVM storage containers for your KVM configured pool named vms.

Here's a good LVM KVM Pool guide from RedHat that includes more information (though it's not as simple as following this guide).

Running PolicyKit (pkexec) Commands without Prompting for Authentication

Saturday, December 8th, 2018

PolicyKit pkexec – Running without Prompting for Authentication

The following guide explains how to configure a pkexec command to run without prompting for authentication.  This is helpful when you want to grant root access to key piece of the system (such as allowing virsh commands from another user when running KVM virtual machines) or just want to run a GUI command as root without having to login or use authentication. 

https://askubuntu.com/questions/383747/how-to-configure-pkexec-to-not-ask-for-password#answer-388660

Or in our own archive in case the above link disappears.