Adding SAS RAID Drivers to CentOS 8 and Red Hat Linux During Installation

CentOS 8 and Red Hat Linux 8 removed a lot of built in RAID controller and SAS drivers.  As such, you'll need to identify your SAS RAID controller card model number, and then during the installation of CentOS 8 or Red Hat, you will need to follow these instructions (modifying them for your hardware).

https://gainanov.pro/eng-blog/linux/rhel8-install-to-dell-raid/

If for some reason the link above is no longer available, I saved and archived a copy which can be read here.

Fixing Sound in Ubuntu

If your sound quits working randomly after installing updates via the apt system (via sudo apt-get update && apt-get upgrade) or via the Software & Updates graphical program, it's possible that some of the drivers have not been installed with the latest kernel updates.

To fix this, try running the below command:

sudo apt install linux-modules-extra-$(uname -r)

Reboot.  Your sound should hopefully work again!

If the above command doesn't work (older versions of Ubuntu do not have this package), please see the generic information here:

https://itsfoss.com/how-to-fix-no-sound-through-hdmi-in-external-monitor-in-ubuntu/

Intel Original Compute Stick – Keep WiFi from Breaking – Block Kernel Updates and Chestersmill-Settings Updates

If you have the original Intel Compute Stick (STCK1A8LFC [1GB of RAM] or STCK1A32WFC [2GB of RAM] models from 2015), you'll need to prevent a few software packages from updating so that these updates won't break your WiFi!  I've never been able to get the WiFi to work with these Intel Compute Sticks running a kernel newer than version 3.16 on any version of Ubuntu.  I've also never been able to get the WiFi to work on anything but Ubuntu 14.04, so you might be stuck having to run this older version of Ubuntu.  Also, updates to the chestersmill-settings package can break your WiFi.  To prevent both scenarios from breaking your WiFi, simply prevent the below packages from being updated.

Preventing packages from being updated can be accomplished using hold statuses in the apt system as explained on AskUbuntu:

https://askubuntu.com/questions/18654/how-to-prevent-updating-of-a-specific-package

Basically, you'll need to run the below commands to prevent WiFi from breaking due to buggy kernel and chestersmill-settings package updates:

sudo apt-mark hold chestersmill-settings
sudo apt-mark hold linux-image-$(uname -r)
sudo apt-mark hold linux-image-generic
sudo apt-mark hold linux-generic

You can now safely update software packages without worrying about your WiFi being broken by kernel and chestersmill-settings package updates!

Get Latest Software from Ubuntu Advantage for Ubuntu 14.04

Ubuntu 14.04 is also still supported via the Extended Security Maintenance (ESM) Ubuntu Advantage program (https://ubuntu.com/advantage).  

To get updated software and packages until April of 2022, you'll need to get an Ubuntu Advantage key.  Get your key using your UbuntuOne account on this page:  https://ubuntu.com/advantage

Now, install the Ubuntu advantage client by using the commands below:

sudo add-apt-repository ppa:ua-client/stable
sudo apt-get update
sudo apt-get install ubuntu-advantage-tools

Set your key using the command below (using your key rather than YOUR_KEY_HERE):

sudo ua attach YOUR_KEY_HERE

Now update and install the upgraded packages available via Ubuntu Advantage:

sudo apt-get update && sudo apt-get upgrade

Do Not Use tcp_tw_recycle

I had a very strange connectivity issue recently that I was only able to reproduce intermittently on my own LAN network when connecting to a few of my servers hosting websites that process and receive tons of simultaneous connections at any point in time.  Basically, my connection to a specific set of websites that I host would timeout from my home internet connection.  However, I was never able to reproduce this issue when connecting to the same sites from other networks belonging to my family and friends. 

From my home connection, I used TCPView and saw that SYN_SENT packets were supposedly sent to my servers to establish a connection.  Unfortunately, the server never replied to some of these requests.  As such, my connection would timeout at times, and work perfectly fine sometimes.  I looked at DD-WRT's connection table, and it also claimed that the packets had been sent, but that they were in an UNREPLIED state when I experienced issues.  Thus, packets were supposedly being sent, but the server was not responding at times.  After spending nearly a week trying to tackle this issue and buying new cable internet equipment (an officially supported Comcast modem), I tracked down the issue, and it ended up being a TCP configuration setting on my servers rather than my home LAN equipment.

Modem or Router's Fault?

Originally, I thought my issue was caused by the DD-WRT open source firmware I was running on my wireless router.  If I restored the router's settings to DD-WRT's factory defaults, I could always connect to the websites I was having intermittent connection timeout issues on.  I suspected it might be my router after trying an older router which didn't have any problems either.  I even upgraded the DD-WRT firmware to the latest version and rebuilt my complicated network configuration from scratch.  Unfortunately, the issue was still there.  Thus, despite mixed results with different routers, I started to wonder if the issue was on my server's end.

Finally Fixed

I started looking at sysctl TCP settings I could adjust on my router, and I ended up comparing some of these values to the ones used on my servers (that were hosting the problem websites).  Eventually, I came across configuration values I had changed myself several months ago which were supposed to help the server support multiple simultaneous connections.

After reading this StackOverflow thread (https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both), I decided I would try disabling the tcp_tw_recycle setting.

/proc/sys/net/ipv4/tcp_tw_recycle was set to 1 (enabled) from tweaks I had run that I had found on the internet.  After I disabled it, /proc/sys/net/ipv4/tcp_tw_recycle was set to 0 (disabled).  By default, Linux keeps tcp_tw_recycle disabled.  Again, this is something I had changed for tuning reasons.  After disabling this setting and rebooting the server, I no longer have any issues connecting to the severs in question.  No more connection timeouts, and everything works properly again.

I have no idea why I wasn't able to reproduce this issue on other networks.  I thought it was my network equipment (modem and router), but it ended up being the server.

Lessons Learned

Be careful when applying settings you find online.  Sometimes, they may not work, or their usage may be buggy.  In fact, net.ipv4.tcp_tw_recycle has been removed from Linux in kernel versions newer than 4.12 by default.  I'm guessing this is because it doesn't work, as I experienced.  Do NOT use  net.ipv4.tcp_tw_recycle! I kept tcp_tw_reuse enabled, so you can enable this setting without running into problems.  Just don't for the love of anything use tcp_tw_recycle!  It doesn't work, and it will cause you headaches trying to track down strange intermittent issues!

Blackbird for Windows (7, 8, and 10)

When it comes to Windows, getting rid of telemetry, keyloggers, and other spyware Microsoft has embedded in your operating system can be rather difficult.  Fortunately, there are a few utilities that can help you take back control over your privacy and security.  One of these utilities is Blackbird, and it is now my preferred privacy and security utility for removing the Microsoft bloatware and spyware that Microsoft has embedded in their latest versions of Windows (Windows 7, Windows 8, and Windows 10). 

To use Blackbird, simply download and run the latest version from their site:

https://www.getblackbird.net/

If for some reason you can't download it from their official site, you can download the latest version from this mirror.

Fix for Mapped Network Drive Issues

After running Blackbird and using it to remove Microsoft's embedded spyware, your mapped network drives to your Network Attached Storage (NAS) drives may no longer work or load properly.  To fix this, download and extract this zip file (named blackbird_fix_smb1_nas_drives.zip) into the same directory where you unzipped the blackbird.exe file.  Then, double click on the "blackbird-network-issues-fix-including-smbv1.bat" file which will run scripts to fix your Server Message Block Version 1 (SMB1) settings.  Reboot your computer after running the batch file, and your NAS drives should work again.

Destroy Windows Spying

I used to use Destroy Windows Spying, but unfortunately, it hasn't been updated in a long time and is no longer being actively developed or maintained, and as such, Blackbird is now my preferred destroy windows spying utility!

C# Binding Redirects – Finding and Detecting Assembly Version of Any DLL

When dealing with binding redirects, you may not know the newVersion value to use for a recently updated library.  The file version of the DLL is not necessarily the assembly version of the DLL, and when it comes to binding redirects, you must use the correct assembly version for a particular library. 

For example, in the web.config for one of our MVC projects we have the following:

<runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
       <dependentAssembly>
         <assemblyIdentity name="Antlr3.Runtime" publicKeyToken="eb42632606e9261f">
         <bindingRedirect oldVersion="0.0.0.0-3.5.0.2" newVersion="3.5.0.2">
       </dependentAssembly>
    </assemblyBiding>
</runtime>

If we update the Antlr3.Runtime package to the latest version in the nuget package manager, our binding redirects may not be automatically updated. As such, we will have to update it manually ourselves with the correct newVersion assembly version value for the updated DLL. To find the assembly version, run this PowerShell script (updating the path to the DLL as necessary):

[Reflection.AssemblyName]::GetAssemblyName('C:\development\bin\Antlr3.Runtime.dll').Version

This is the version number that is needed for the updated binding redirect in the "newVersion" attribute.

Dell c1100 CS24-TY Latest BIOS and BMC Firmware Files

Download the latest BIOS and BMC Firmware Files for the Dell c1100 CS24-TY 1U Server

The file above contains instructions and guides for updating both the BIOS and BMC Firmware to the latest released versions for official Dell c1100 CS24-TY servers and unofficial DCS or Quanta c1100 versions.

For Quanta or DCS servers, follow this guide:

https://johannes.skartland.net/2015/11/flashing-c1100/  | Archived Version

Linux KVM:  Using Multiple NICs and Routing Traffic Properly Between Them

When setting up a KVM guest to use multiple network interface controllers (NICs), additional ip routes may be needed in order for the additional interfaces to work properly.  For example, if you configure a NIC with a public static IP address and a NIC with an internal private DHCP LAN IP address, you must create a route for any traffic that comes through the DHCP LAN IP address to respond via the interface from which the request originated.  Otherwise, forwarded NAT traffic from the main KVM host to the DHCP internal LAN IP will reach its destination, but no response will be sent back (because it will attempt to send the response via the configured static IP address interface which may NOT be the original destination of the senders request).

The Solution:

https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming/23345#answer-23345

From the above link, the solution for me was to do the following in the KVM guest virtual machine:

Only needs to be done once:

sudo -i
echo 200 isp1 >> /etc/iproute2/rt_tables

Setting up the route (adjust variables as necessary):

sudo -i
ip rule add from <interface_IP> table isp1 priority 900
ip rule add from <interface_IP> dev <interface> table isp1
ip route add default via <gateway_IP> dev <interface> table isp

The command I used for my specific setup:

sudo -i
ip rule add from 192.168.122.10 table isp1 priority 900 
ip rule add from 192.168.122.10 dev ens9 table isp1 
ip route add default via 192.168.122.1 dev ens9 table isp1

Making it permanent (apply on system start up):

sudo -i
nano /etc/network/interfaces

I added the below post-up rules (adjust variables as necessary):

auto ens9
iface ens9 inet dhcp
        post-up ip rule add from <interface_IP> table isp1 priority 900
        post-up ip rule add from <interface_IP> dev <interface> table isp1
        post-up ip route add default via <gateway_IP> dev <interface> table isp1

The route is created whenever the dhcp interface is brought up.

Obtaining Let's Encrypt HTTP Validation Server IP Addresses

Use your webserver logs:

sudo apt-get install john
cat access_log.1 | grep "Let's Encrypt" | awk '{print $1}' | unique ips
cat ips

Recreating a Lost or Removed EFI Windows Boot Partition

In case you remove an SSD with a Windows Boot Partition that boots an installation of Windows, you'll need to recreate the boot system to be able to successfully boot again.  To recreate the boot EFI partition on another drive or partition, perform the following:

1) Create or use an existing Windows installation disc or flash drive (you can use Rufus USB to format a flash drive with a Windows ISO)
2) Boot PC using your Windows installation media (a CD, DVD, flash drive, etc containing Windows installation files)
3) Press SHIFT + F10 on the first screen to bring up Command Prompt
4) Run the following commands and click Enter each time at Command Prompt:

diskpart
list disk
select disk N (N refers to the disk which contains the deleted the EFI System partition)
list partition
create partition efi size=200
format quick fs=fat32
list partition
list volume (find the volume letter which belongs to the installed Windows OS)
exit (exit diskpart)
bcdboot M:\windows (M refers to the volume letter of installed Windows OS)

Add any additional Windows installations by repeating the bcdboot command followed by the installation path for any other Windows installations you want added to the menu.  For example, I added my Windows 7 partition using the below command:

bcdboot P:\windows

You should now be able to boot Windows again without needing the old boot partition or drive.

Instructions were modified from this post.