Archive for the ‘Guides’ Category

Running and Debugging Specific WordPress Cron Hooks Manually Using WP-CLI

Thursday, December 22nd, 2022

Running and Debugging Specific WordPress Cron Hooks Manually Using WP-CLI

If your WordPress instance is configured with debugging turned off (WP_DEBUG set to false in wp-config.php) and you need to troubleshoot, debug, or run a specific WordPress cron hook or function manually, you'll find that it's pretty tough to do so.  Fortunately, I found a relatively easy way to do exactly this using WP-CLI.

Install WP-CLI

Install WP-CLI by using the below commands as root:

sudo -i
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
chmod +x wp-cli.phar
sudo mv wp-cli.phar /usr/local/bin/wp

Now, change into your WordPress instance vhosts directory:

cd /var/www/vhosts/{YOUR_DOMAIN}/httpdocs

Update the above path as necessary.

To run all the scheduled WordPress cron events, use this command:

wp cron event run --all --allow-root

To run a specific WordPress cron hook, use this command:

wp cron event run "NAME_OF_CRON_HOOK" --allow-root

If you need help determining the name of the cron job hook you specifically want to run or debug, install the WP Control plugin in your WordPress instance. Then visit the /wp-admin/tools.php?page=crontrol_admin_manage_page page to see a list of cron hooks and when they run.  Grab the name from the "Hook" column and update your wp cli command to run that cron event!

You can add echo statements and other code to your individual cron function plugins to better debug and troubleshoot them.

OpenVPN Expired CRL – VPN Won’t Connect

Wednesday, December 7th, 2022

OpenVPN Expired CRL – VPN Won't Connect

Recently, I ran into an issue where OpenVPN was no longer working for existing clients.  After looking at the OpenVPN log in /var/log/openvpn.log, I found the following:

VERIFY ERROR: depth=0, error=CRL has expired:

If you see an OpenVPN error about an expired certificate revocation list (CRL), here's how to generate a new CRL:

cd /etc/openvpn/easy-rsa
EASYRSA_CRL_DAYS=3650 ./easyrsa gen-crl
cp /etc/openvpn/easy-rsa/pki/crl.pem /etc/openvpn/crl.pem
chown nobody:nogroup /etc/openvpn/crl.pem
service openvpn restart

Problem solved!

Support Older TLS Versions in Newer Ubuntu / Debian OS Versions

Monday, December 5th, 2022

Support Older TLS Versions in Newer Ubuntu / Debian OS Versions

Edit openssl.conf file:

sudo nano /etc/ssl/openssl.cnf

Add this line at the top:

openssl_conf = openssl_init

And add these lines at the end:

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT@SECLEVEL=1

https://askubuntu.com/questions/1233186/ubuntu-20-04-how-to-set-lower-ssl-security-level#answer-1296578

Ubuntu: Allow Automatic Updates for Specific Packages Only

Tuesday, June 14th, 2022

Ubuntu: Allow Automatic Updates for Specific Packages Only

If you want to allow Google products and packages to update automatically, follow this guide.

You can also add additional sources that should update automatically following the same process.

This is helpful when using Selenium, WebDriver for Chrome, and Python.  Doing this allows you to always use the most up-to-date version of all of these dependent packages.

Tested in Ubuntu 20.04

Setup Remote Logging on an Ubuntu rsyslog Server for DD-WRT to Use

Wednesday, June 9th, 2021

Setup Remote Logging on an Ubuntu rsyslog Server for DD-WRT to Use

Enable remote logging on an Ubuntu server by configuring rsyslog to allow remote connections from port 514 (adjust as needed):

sudo nano /etc/rsyslog.conf

Uncomment the imudp and imtcp load module statements like so (adjusting as needed):

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Create a logging template and apply it only to remote hosts that start with "c-" (comcast connection remote host prefix [followed by the IP address of the device which can change])

# Comcast remote logging
$template remote-incoming-logs, "/var/log/remote_logs/%HOSTNAME%/%PROGRAMNAME%.$
if $fromhost startswith "c-" then -?remote-incoming-logs

Save and quit.

Restart the rsyslog daemon:

sudo service rsyslog restart

Remote logs will be stored in /var/log/remote_logs

Configure logrotate to process and rotate these logs automatically (so you don't lose them and have a history on them):

sudo nano /etc/logrotate.d/ddwrt

Paste these contents into the file:

/var/log/remote_logs/*.log /var/log/remote_logs/*/*.log {
    daily
    missingok
    compress
    delaycompress
    su syslog adm
}

Save and quit.

Everything has been configured, and remote logging should work from your DD-WRT router once you set the remote URL to your server's IPAddress:port combo and apply the changed settings.

CentOS LVM and Software RAID Partitioning Instructions

Sunday, May 30th, 2021

Installing and Configuring CentOS to Host KVM Virtual Machines

GUI

When configuring a fresh install of CentOS for a KVM host machine (the main server that hosts all of the virtual machines), I like to run a GUI to make managing some of the virtual machines easier.  Thus, during install, choose the options for CentOS with Minimal GUI:

RAID 10 LVM Partitions

When configuring the hard drive partitions, set it up to use RAID 10 LVM SOFTWARE RAID:

Create volume group called "vms" without the quotes that is setup as RAID 10 (set volume group space to be as large as possible).

Set the "/" partition to 100GB XFS LVM (RAID10).

Set the "swap" partition to 32GB.

Only setup those two partitions.  The remaining space in the RAID 10 volume group "vms" will be used for KVM containers (and the remaining space does NOT need to be assigned to any mount points).

That's all.

Increasing KVM Guest Hard Disk (Hard Drive) Space

Sunday, May 30th, 2021

Increasing KVM Guest Hard Disk (Hard Drive) Space

Increasing the hard drive space in a KVM guest can be rather tricky.  The first step is to shutdown (completely turn off) the guest machine by running the below command from the guest system:

sudo shutdown -h now

Once the guest machine has been turned off (verify it is off by using sudo virt-manager on the host machine to see if it's no longer running), on the host machine, resize the LVM partition by running the following command (and adjust the size as necessary):

sudo lvextend -L+78G /dev/vg_vps/utils

If you need help identifying the name of the disk your guest has been assigned, run this command from the host:

sudo virsh domblklist {VIRSH_NAME_OF_VIRTUAL_MACHINE}

For my example, I would use this command:

sudo virsh domblklist utils

From the host machine, download the GParted live ISO image for your system's architecture (x86 or x64).  Start virt-manager:

sudo virt-manager

Assign a CD drive to the virtual machine you're expanding the hard drive space for, and assign / mount the GParted ISO to it.  Change the boot order so that the KVM guest boots from the CD first.  Save your settings and start the KVM guest virtual machine.  Boot into GParted Live.  GParted will run automatically.  Use GParted to expand the partitions so that they make use of the added storage based on your own preferences.  Apply the resize operation.  Exit GParted and shutdown the virtual machine so that it's off again. Remove the CD drive from the boot options from virt-manager, and then start the KVM guest again. 

If Guest Doesn't Use LVM Partitioning

If your KVM guest virtual machine hasn't been configured to use LVM, the added hard drive space should already be available to your system.  Verify it has been expanded by again running the df -h command.  You're done!

If Guest Uses LVM

Let the OS boot.  From the guest, the file system needs to be resized itself.  You can do this by running the following command to see the current space allocated to your system's partitions:

df -h

You'll see a bunch of output similar to:

Filesystem                  Size  Used Avail Use% Mounted on
udev                        2.9G     0  2.9G   0% /dev
tmpfs                       597M  8.3M  589M   2% /run
/dev/mapper/utils--vg-root  127G   24G   98G  20% /
tmpfs                       3.0G     0  3.0G   0% /dev/shm
tmpfs                       5.0M     0  5.0M   0% /run/lock
tmpfs                       3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/vda1                   720M   60M  624M   9% /boot
tmpfs                       597M     0  597M   0% /run/user/1000

You'll notice that the added hard drive space doesn't show up on any of the partitions.  However, it is available to be assigned to these partitions.  To assign additional space, you will need to resize it using these commands (run from the guest virtual machine… the machine you're resizing):

lvextend /dev/mapper/utils--vg-root -L +78G
resize2fs /dev/mapper/utils--vg-root

Obviously, you need to substitute the name of the LVM partition with the one from your system shown in your output of the df -h command.

Resources

https://tldp.org/HOWTO/LVM-HOWTO/extendlv.htmlMirror if Offline

https://sandilands.info/sgordon/increasing-kvm-virtual-machine-disk-using-lvm-ext4Mirror if Offline

Obtaining Let’s Encrypt HTTP Validation IP Addresses

Saturday, July 11th, 2020

Obtaining Let's Encrypt HTTP Validation Server IP Addresses

Use your webserver logs:

sudo apt-get install john
cat access_log.1 | grep "Let's Encrypt" | awk '{print $1}' | unique ips
cat ips

Installing Chrome WebDriver (Linux Script)

Wednesday, August 28th, 2019

Installing Chrome WebDriver (Linux Script)

Find out which version of Chrome is installed on your system before running the below commands.  You can find out your chrome version by running the following command:

google-chrome --version

Adjust the version number (replace {VERSION_NUMBER})  in the below commands to match the version installed on your system!!!

sudo -i
cd ~/Downloads
rm chromedriver_linux64.zip
wget -N https://chromedriver.storage.googleapis.com/{VERSION_NUMBER}/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver

Selenium and other libraries that rely on the Chrome WebDriver should now work properly.

CentOS – Using NAT with KVM Guests

Wednesday, July 17th, 2019

CentOS – Using NAT with KVM Guests

Please note that all commands in this guide must be run on the main HOST machine (the physical machine).  They should not be run on KVM guests (virtual machines).

If your server has a limited number of IPv4 addresses, it might be best to setup and run some virtual machines that are configured to use the default NAT network interface that KVM provides.  This will allow you to run multiple virtual machines that share the same IP address.  Think of it as setting up a home network with multiple devices with certain ports forwarded to specific devices for incoming connections.

First, create the virtual machines that are going to use NAT using virt-manager as you normally would.  In the virtual machine configuration wizard, assign the default NAT network interface named "virbr0".  After the virtual machines have been created, shut down the virtual machines.  Now, we'll assign these virtual machines static LAN IP addresses so that we can port forward certain ports and always have them reach the proper virtual machines. 

The first thing we need to do is get the MAC address of each virtual machine.  Write down the name of the virtual machine and its MAC address, as we'll need this information later on when we edit the NAT interface and assign static LAN IP addresses to our virtual machines.  Run this command to retrieve the MAC address for a specific virtual machine.

virsh dumpxml VM_NAME | grep -i '<mac'

Using the MAC address information from the VMs we want to use NAT with, edit the default NAT interface by running the below command. 

virsh net-edit default

If for some reason the NAT interface is not named default, you can find it by running the below command:

virsh net-list

After the <range /> entry, assign the static LAN IP addresses similar to the following:

<dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:ff:4a:2a' name='vm1' ip='192.168.122.13'/>
      <host mac='52:54:00:bb:35:67' name='vm2' ip='192.168.122.14'/>
      <host mac='52:54:00:aa:d9:f2' name='vm3' ip='192.168.122.15'/>
</dhcp>

Save the file with your desired values and quit the editor.  Restart the NAT interface by running the below commands:

virsh net-destroy default
virsh net-start default

Now, you'll need to setup your iptables port forwarding rules.  Adjust the below rules as necessary (changing the port numbers to the ones you want to use) and then save them so that they persist:

iptables -I FORWARD -o virbr0 -d 192.168.122.13 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39989 -j DNAT --to 192.168.122.13:39989
iptables -I FORWARD -o virbr0 -d 192.168.122.14 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39990 -j DNAT --to 192.168.122.14:39990
iptables -I FORWARD -o virbr0 -d 192.168.122.15 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 39991 -j DNAT --to 192.168.122.15:39991
iptables -t nat -A POSTROUTING -s 192.168.122.0/24 -j MASQUERADE
iptables -A FORWARD -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i virbr0 -o br0 -j ACCEPT
iptables -A FORWARD -i virbr0 -o lo -j ACCEPT
iptables-save
service iptables save

Congrats, your virtual machines are now using NAT, have been assigned static LAN IP addresses, and iptables rules on the host server have been configured to port forward specific ports to each NAT VM.


Persistently Saving NAT Port Forward Rules

The only solution I found that would persistently save my NAT forwarding rules is to create a libvirt hook bash script as mentioned here

service iptables stop
iptables -F
service iptables save
service iptables start
mkdir -p  /etc/libvirt/hooks
nano  /etc/libvirt/hooks/qemu

The contents of the "/etc/libvirt/hooks/qemu" file should look similar to the following:

#!/bin/bash
# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "vm1" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.13
   GUEST_PORT=39989
   HOST_PORT=39989   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
    /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
    /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm2" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.14
   GUEST_PORT=39990
   HOST_PORT=39990   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi
if [ "${1}" = "vm3" ]; then   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.15
   GUEST_PORT=39991
   HOST_PORT=39991
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

Save and exit.  Make the script executable.

Here is an example of forwarding multiple ports with one entry:
 

if [ "${1}" = "vm_name" ]; then
   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.3
   GUEST_PORTS=(15231 15232 15233)   
   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        for i in ${GUEST_PORTS[@]}; do
            /sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
            #/sbin/iptables -t nat -D PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            /sbin/iptables -t nat -D PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -D PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -D PREROUTING -p all -j DNAT --to $GUEST_IP:${i}
        done
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        for i in ${GUEST_PORTS[@]}; do
            /sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
            #/sbin/iptables -t nat -I PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            /sbin/iptables -t nat -I PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -I PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
            #/sbin/iptables -t nat -I PREROUTING -p all -j DNAT --to $GUEST_IP:${i}
        done
   fi
fi
chmod +x /etc/libvirt/hooks/qemu

If you're attempting to forward incoming traffic (only traffic originating from the outside), be sure to exclude the main virbr0 interface so that requests originating from the DHCP LAN aren't forwarded to itself.  This is helpful for port forwarding DNS (port 53).  Incoming requests from outside of the network will be forwarded to the specified server, but outgoing DNS requests made from the server will NOT be forwarded back to itself.  Basically, you want to forward all traffic on port 53 to that specific server except for traffic that originates from that server.  So, be sure to add this line if that is the case:

/sbin/iptables -t nat -I PREROUTING \! -i virbr0 -p udp --dport 53 -j DNAT --to $GUEST_IP:$GUEST_PORT

More information can be found here:  https://serverfault.com/questions/1042383/kvm-port-forwarding-port-53-to-guest-via-nat-temporary-failure-in-name-resolut/1042394#1042394

Reboot the host server.


Old Instructions for Persistent Saving (Non-Working)

If your iptables forwarding rules are not persisted after the host machine is rebooted or shutdown, run the following commands:

sudo -i
yum install -y iptables-services
systemctl stop firewalld
systemctl disable firewalld
systemctl enable iptables
nano /etc/sysconfig/iptables-config 

Change the below values to "yes":

IPTABLES_SAVE_ON_RESTART="yes"
IPTABLES_SAVE_ON_STOP="yes"

Save and exit.  Reboot the server.

If you're still having issues, try this (will clear your existing iptables rules):

iptables-save > iptables_bk
service iptables stop
iptables -F
<run iptables NAT rules here>
<run any other iptables rules you want>
service iptables save
service iptables start

More Detailed Guide