TCP Syn Flood Hardening – Kernel IPv4 Settings

With the increasing TCP flood attacks and AI bot scraping floods, here are some optimizations that should be added to the end of your /etc/sysctl.conf file:

# Hardening
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.tcp_max_syn_backlog=2048
net.ipv4.tcp_synack_retries=2
net.ipv4.conf.all.accept_redirects=0

Reboot for the settings to take effect.

Dell C1100 – Partitioning for RAID 10

In the Rocky Linux installer, choose custom partitioning.  First, create a /boot mount point that is 2GiB in size.  Set device type to RAID and RAID Level to RAID1.  Next, create / as an LVM partition on a RAID 10 pool.  Lastly, create the SWAP area as an LVM partition on the RAID 10 pool.  That's all you need to do. 

Grub will be synced across the four drives.  Thus, if one drive fails, the other drives can still boot Rocky Linux so that the RAID array can be rebuilt.   

Clone and Compress LVM Volume to Disk Image

If you need to backup an LVM volume, you can do so by first turning off the virtual machine that is using the volume, and then you can use the dd and gzip utilities to create a copy of the volume to an .img file and compress the image file.  This compressed image file can then be transferred remotely over SSH or HTTP (for fastest results) to another server where it can be restored.

Need More Storage Space for the Backup Image (Optional)?

If you don't have enough storage space on the root LVM volume the main system uses on the host server to handle saving the backup image file to, you can always create a new LVM and then mount it on the host LVM file system to save it there.

These steps are optional:

1.  Create new hard drive in your vm pool using virt-manager and name it temp_storage – you don't need to setup the OS or continue creating the virtual machine

2.  Format and mount this new LVM volume (update vms with your pool's actual name – on my server, it is named vms):

mkfs.ext4 /dev/mapper/vms-temp_storage
mount /dev/mapper/vms-temp_storage /mnt/store

Copy and Compress the LVM

Use the below commands to copy your LVM volume to an img file.  Update the names and the paths as needed.

dd if=/dev/vms/name_of_lvm of=/mnt/store/name_of_lvm.img bs=8M status=progress
gzip name_of_lvm.img -1

Uncompressing and Restoring the LVM

Just decompress the image file and then use the dd command to restore the image to the LVM volume created on the target or restore server.

gunzip name_of_lvm.img.gz
dd if=name_of_lvm.img of=/dev/vms/restore-lvm-name bs=8M status=progress

And you're done 🙂

Using Wireless Adapter as KVM Network Bridge (br0 or br1) – Rocky Linux 9.X

They said it couldn't be done, but it can, depending on your wireless adapter card.  I recently bought a Minisforum EM680 Mini PC, and to my surprise, it didn't come with any ethernet ports.  It only has a Wi-Fi PCI-E card in it.  Ok, so that presents a problem, as I wanted to use it as a KVM virtual machine host with bridged networking.  At first, I wasn't able to get this to work, but after finding this post, I got it working!  If your wireless adapter supports 4addr (WDS mode), you can get it working as you normally would when using an ethernet port.  Here's how.

Step 1:  Connect to your desired wireless network normally post installation of Rocky Linux 9.

Step 2:  Create your network bridge configurations as you normally would (not specified in this guide).  My script creates br0 (for use at datacenters with static IPs) and br1 (a general DHCP bridge). 

Step 3:  Enable 4addr for your adapter.

Get the name of your wireless device by running the following command:

ip a

Mine happens to be named wlp1s0. Yours could be different. Make a note of the name of your connected wireless adapter and use it in the below scripts:

sudo nano /etc/systemd/system/4addr.service 

Add the following contents to this file:

[Unit]
Description=wlan-4addr
Wants=network.target
Before=network.target systemd-networkd.service
BindsTo=sys-subsystem-net-devices-wlp1s0.device
After=sys-subsystem-net-devices-wlp1s0.device

[Service]
Type=oneshot
ExecStart=/usr/sbin/iw dev wlp1s0 set 4addr on
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Save and exit.

Enable the service at boot:

sudo systemctl enable 4addr.service 

Step 4:  Reboot

Step 5:  Check that 4addr is on and working

iw dev wlp1s0 info

You should see something like:

4addr: on 

At the end of the output. 

Step 6:  Configure your wireless adapter to be controlled by your desired network bridge.  In my case, I wanted my wireless adapter to be a part of br1 (my general DHCP bridge).  To do this, I edited the associated .nmconnection file in the /etc/NetworkManager/system-connections directory.  The name of the .nmconnection file will be the wireless SSID you're connected to.  So, for example, if your SSID is test, it will be named test.nmconnection.

Add the controller and port-type configuration lines as shown in the below sample configuration.  My .nmconnection configuration looks like this (SSID and passwords removed):

[connection]
id=cool
uuid=015a8f8f-5440-40b5-8caf-e6ff3f9d63e1
type=wifi
interface-name=wlp1s0
controller=br1
port-type=bridge

[wifi]
mode=infrastructure
ssid={WIFI_SSID_HERE}

[wifi-security]
auth-alg=open
key-mgmt=wpa-psk
psk={WIFI_PASSWORD_HERE}

[bridge-port]

Save your changes.

My ifcfg-br1 bridge configuration file in the /etc/sysconfig/network-scripts directory looks like this:

TYPE=Bridge
BOOTPROTO=dhcp
DEVICE=br1
ONBOOT=yes
ZONE=public

Step 7:  Reboot

KVM guests using br1 and the default NAT interface (managed by QEMU and KVM) will work just fine via your wireless device now!

Dual Boot Linux (Ubuntu 22.04) and Windows 11 on Modern Systems – UEFI

In order to setup a dual boot of Windows 11 and Ubuntu 22.04 on a modern system that uses UEFI, follow these steps.

1. Install Windows 11 first leaving some unpartitioned space (at least 60GB is my recommendation) on the drive you're installing Windows on.
2. Boot up the Ubuntu installer.
3. During installation, you'll be presented with an Installation Type options screen.  Choose "Something else". 

4. On the next screen, you'll see a list of drives and partitions.  On the same drive you installed Windows, create 3 new partitions. 
   1. Create an EXT4 partition for the / mount point at least 40GB in size (this is the main drive for Linux files).
   2. Create a SWAP partition at least 18GB in size.
   3. Create an EFI partition at least 500MB in size.  This is extremely important in order to get grub to install properly. 
5. Leave the "Device for boot loader installation" set as the top level drive that Windows and Ubuntu was / is being installed on.  You should not select an individual partition here.

6. Complete the installation process. 
7. You might need to change the UEFI boot order in the BIOS of your system to boot Ubuntu / Linux first versus booting the Windows EFI partition.  Since you created an EFI partition for your Linux install, it should show up as a bootable option in the bios.  Set / adjust accordingly.
8. That's it!

RAID Synchronization CRON Job Affecting Performance

For some FakeRaid configurations, CentOS 7 and newer variants may run a RAID synchronization job configured in the /etc/cron.d directory in a file named raid-check.

This job is responsible for making sure the RAID array is in sync across all drives.  It runs by default every week on Sunday at 1 AM.

# Run system wide raid-check once a week on Sunday at 1am by default

However, this was not a convenient time for my users, as they were gaming at this time, so rather than affect server performance, I changed the cronjob to:

0 5 1 * * root /usr/bin/test $(/usr/bin/date +\%u) -ne 6 && /usr/sbin/raid-check

Thus, the sync job now runs once a month on the 1st at 5 AM.  And, it will not run if the day of the week is a Saturday.  This applies to several of my C1100 servers.

Manage Dell C1100 CS24-TY Servers Using IPMI Tool

Dell C1100 servers can be managed using the IPMI tool which interfaces with the low level BMC interface.  Dell has also released a bmc specific utility that compliments the generic ipmitool to better configure and manage specific settings that are difficult to configure using ipmitool on its own.

Here's how to use both ipmitool and the bmc utilities.  First, login as root:

sudo -i

Now, install the ipmitool:

yum install OpenIPMI ipmitool -y

Start the ipmi service:

service ipmi start

Download the bmc utility:

mkdir -p ~/Downloads/bmc
cd ~/Downloads/bmc
wget -N "http://dinofly.com/files/dell_c1100/bmc-2014-10-15.zip"

If your system is running versions of PERL 5.26 or newer, download this version instead:

# For PERL 5.26+
wget -O "bmc-2014-10-15.zip" -N "http://dinofly.com/files/dell_c1100/bmc-2014-10-15_perl_5.26.zip"

Unzip the bmc tool and make it executable:

unzip bmc-2014-10-15.zip
chmod +x bmc

Here's how to change the BMC interface IP settings:

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr x.x.x.x
ipmitool lan set 1 netmask x.x.x.x
ipmitool lan set 1 defgw ipaddr x.x.x.x
./bmc nic_mode set [dedicated|shared]
./bmc allinfo

If the BMC web interace is not responding, try this:

ipmitool bmc reset cold

Using Serial Over Lan SOL

If your system fails to boot and you are booted into an emergency shell and need to copy log files off of the remote system that is stored in ram and can't mount your file system, you can transfer the information using sol

On a linux laptop or computer, just install ipmitool, and then issue the following commands:

On the laptop or computer:

ipmitool -I lanplus -H [BMC_IP] -U [USER] -P [PASS] sol deactivate
ipmitool -I lanplus -H [BMC_IP] -U [USER] -P [PASS] sol set volatile-bit-rate 115.2 1

On the Virtual KVM (launched from Dell's Remote BMC web interface):

stty -F /dev/ttyS1 115200

On the laptop or computer:  

ipmitool -I lanplus -H [BMC_IP] -U [USER] -P [PASS] sol activate > my_log.txt

On the Virtual KVM (launched from Dell's Remote BMC web interface):

cat /run/initramfs/rdsosreport.txt > /dev/ttyS1

That will send the contents via sol to the client machine so you can upload the logs in a bug report later.  Super useful information I've learned!

Running bmc reset cold Before Reboot:

To make sure your system is always reachable in case the BMC is flooded or times out serving the web interface, make sure ipmitool bmc reset cold is run before system reboot.

sudo nano /etc/systemd/system/bmc-reset.service 

Paste the following in:

[Unit]
Description=Reset BMC on Reboot
DefaultDependencies=no
Before=reboot.target

[Service]
Type=oneshot
ExecStart=/usr/bin/ipmitool bmc reset cold
ExecStart=/usr/bin/sleep 120
RemainAfterExit=yes

[Install]
WantedBy=reboot.target

Enable the service:

sudo systemctl daemon-reload
sudo systemctl enable bmc-reset.service 

Replace Hard Drive in a RAID 10 Array and Sync the RAID 10 Array to the New Hard Drive

I had the hardest time rebuilding a RAID 10 array after replacing a hard drive.  I didn't fail the old hard drive before removing it from the array, and sometimes, this may not be an option.  What happened in my case is the data center replaced the hard drive that I had shipped to them directly from an eBay seller.  I was hoping that the RAID array would rebuild itself onto the new drive (as I have seen happen before in some circumstances).  However, that may not happen if the replacement drive still has its old RAID array or partition information present, and then, it might be difficult to actually get the RAID array to sync to the new drive. 

In my case, I run LVM (Logical Volume Manager) for my partitions.  This complicates the RAID setup, and I found that mdadm commands didn't work as expected.  If this situation occurs, it is best to boot Rocky Linux or CentOS in recovery mode using a Rocky Linux ISO or CentOS ISO.  Once the recovery system loads, drop to a shell without mounting any file systems.  Next, you will need to deactivate your LVM volume group:

vgdisplay
vgchange -a n my_volume_group # deactivate

Next, examine your md RAID array by running the following command:

cat /proc/mdstat

After running that command, I identied my RAID devices as md126 and md127.  /dev/md127 is considered the parent even though /dev/md126 is where everything is. 

I can get more information about the RAID array by running the below commands:

mdadm --detail /dev/md126
mdadm --detail /dev/md127

Let's fail and remove any removed (no longer existing) drives using this command:

mdadm /dev/md126 --remove failed
mdadm /dev/md126 --remove detached
mdadm /dev/md127 --remove failed
mdadm /dev/md127 --remove detached

Next, we need to identify the hard drive we want to add / replace the removed drive in the array:

lsblk

From running the above command, I noticed that the new drive was /dev/sde, so I needed to wipe its old RAID configuration (if there is any) and then add it to the RAID array.

wipefs /dev/sde
mdadm --add /dev/md127 /dev/sde

Check to see if the syncing process has started:

cat /proc/mdstat

You may or may not need to run the below command to get the RAID device to start syncing to the new drive:

mdadm --grow /dev/md126 --raid-devices=4

Helpful Links:

https://delightlylinux.wordpress.com/2020/12/22/how-to-remove-a-drive-from-a-raid-array/
https://serverfault.com/questions/554553/how-to-delete-removed-devices-from-a-mdadm-raid1
https://unix.stackexchange.com/questions/53129/dev-md127-refuses-to-stop-no-open-files
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/4/html/cluster_logical_volume_manager/vg_activate
https://serverfault.com/questions/676638/mdadm-drive-replacement-shows-up-as-spare-and-refuses-to-sync
https://serverfault.com/questions/554553/how-to-delete-removed-devices-from-a-mdadm-raid1