Do Not Use tcp_tw_recycle

I had a very strange connectivity issue recently that I was only able to reproduce intermittently on my own LAN network when connecting to a few of my servers hosting websites that process and receive tons of simultaneous connections at any point in time.  Basically, my connection to a specific set of websites that I host would timeout from my home internet connection.  However, I was never able to reproduce this issue when connecting to the same sites from other networks belonging to my family and friends. 

From my home connection, I used TCPView and saw that SYN_SENT packets were supposedly sent to my servers to establish a connection.  Unfortunately, the server never replied to some of these requests.  As such, my connection would timeout at times, and work perfectly fine sometimes.  I looked at DD-WRT's connection table, and it also claimed that the packets had been sent, but that they were in an UNREPLIED state when I experienced issues.  Thus, packets were supposedly being sent, but the server was not responding at times.  After spending nearly a week trying to tackle this issue and buying new cable internet equipment (an officially supported Comcast modem), I tracked down the issue, and it ended up being a TCP configuration setting on my servers rather than my home LAN equipment.

Modem or Router's Fault?

Originally, I thought my issue was caused by the DD-WRT open source firmware I was running on my wireless router.  If I restored the router's settings to DD-WRT's factory defaults, I could always connect to the websites I was having intermittent connection timeout issues on.  I suspected it might be my router after trying an older router which didn't have any problems either.  I even upgraded the DD-WRT firmware to the latest version and rebuilt my complicated network configuration from scratch.  Unfortunately, the issue was still there.  Thus, despite mixed results with different routers, I started to wonder if the issue was on my server's end.

Finally Fixed

I started looking at sysctl TCP settings I could adjust on my router, and I ended up comparing some of these values to the ones used on my servers (that were hosting the problem websites).  Eventually, I came across configuration values I had changed myself several months ago which were supposed to help the server support multiple simultaneous connections.

After reading this StackOverflow thread (https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both), I decided I would try disabling the tcp_tw_recycle setting.

/proc/sys/net/ipv4/tcp_tw_recycle was set to 1 (enabled) from tweaks I had run that I had found on the internet.  After I disabled it, /proc/sys/net/ipv4/tcp_tw_recycle was set to 0 (disabled).  By default, Linux keeps tcp_tw_recycle disabled.  Again, this is something I had changed for tuning reasons.  After disabling this setting and rebooting the server, I no longer have any issues connecting to the severs in question.  No more connection timeouts, and everything works properly again.

I have no idea why I wasn't able to reproduce this issue on other networks.  I thought it was my network equipment (modem and router), but it ended up being the server.

Lessons Learned

Be careful when applying settings you find online.  Sometimes, they may not work, or their usage may be buggy.  In fact, net.ipv4.tcp_tw_recycle has been removed from Linux in kernel versions newer than 4.12 by default.  I'm guessing this is because it doesn't work, as I experienced.  Do NOT use  net.ipv4.tcp_tw_recycle! I kept tcp_tw_reuse enabled, so you can enable this setting without running into problems.  Just don't for the love of anything use tcp_tw_recycle!  It doesn't work, and it will cause you headaches trying to track down strange intermittent issues!

Blackbird for Windows (7, 8, and 10)

When it comes to Windows, getting rid of telemetry, keyloggers, and other spyware Microsoft has embedded in your operating system can be rather difficult.  Fortunately, there are a few utilities that can help you take back control over your privacy and security.  One of these utilities is Blackbird, and it is now my preferred privacy and security utility for removing the Microsoft bloatware and spyware that Microsoft has embedded in their latest versions of Windows (Windows 7, Windows 8, and Windows 10). 

To use Blackbird, simply download and run the latest version from their site:

https://www.getblackbird.net/

If for some reason you can't download it from their official site, you can download the latest version from this mirror.

Fix for Mapped Network Drive Issues

After running Blackbird and using it to remove Microsoft's embedded spyware, your mapped network drives to your Network Attached Storage (NAS) drives may no longer work or load properly.  To fix this, download and extract this zip file (named blackbird_fix_smb1_nas_drives.zip) into the same directory where you unzipped the blackbird.exe file.  Then, double click on the "blackbird-network-issues-fix-including-smbv1.bat" file which will run scripts to fix your Server Message Block Version 1 (SMB1) settings.  Reboot your computer after running the batch file, and your NAS drives should work again.

Destroy Windows Spying

I used to use Destroy Windows Spying, but unfortunately, it hasn't been updated in a long time and is no longer being actively developed or maintained, and as such, Blackbird is now my preferred destroy windows spying utility!

C# Binding Redirects – Finding and Detecting Assembly Version of Any DLL

When dealing with binding redirects, you may not know the newVersion value to use for a recently updated library.  The file version of the DLL is not necessarily the assembly version of the DLL, and when it comes to binding redirects, you must use the correct assembly version for a particular library. 

For example, in the web.config for one of our MVC projects we have the following:

<runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
       <dependentAssembly>
         <assemblyIdentity name="Antlr3.Runtime" publicKeyToken="eb42632606e9261f">
         <bindingRedirect oldVersion="0.0.0.0-3.5.0.2" newVersion="3.5.0.2">
       </dependentAssembly>
    </assemblyBiding>
</runtime>

If we update the Antlr3.Runtime package to the latest version in the nuget package manager, our binding redirects may not be automatically updated. As such, we will have to update it manually ourselves with the correct newVersion assembly version value for the updated DLL. To find the assembly version, run this PowerShell script (updating the path to the DLL as necessary):

[Reflection.AssemblyName]::GetAssemblyName('C:\development\bin\Antlr3.Runtime.dll').Version

This is the version number that is needed for the updated binding redirect in the "newVersion" attribute.

Dell c1100 CS24-TY Latest BIOS and BMC Firmware Files

Download the latest BIOS and BMC Firmware Files for the Dell c1100 CS24-TY 1U Server

The file above contains instructions and guides for updating both the BIOS and BMC Firmware to the latest released versions for official Dell c1100 CS24-TY servers and unofficial DCS or Quanta c1100 versions.

For Quanta or DCS servers, follow this guide:

https://johannes.skartland.net/2015/11/flashing-c1100/  | Archived Version

Linux KVM:  Using Multiple NICs and Routing Traffic Properly Between Them

When setting up a KVM guest to use multiple network interface controllers (NICs), additional ip routes may be needed in order for the additional interfaces to work properly.  For example, if you configure a NIC with a public static IP address and a NIC with an internal private DHCP LAN IP address, you must create a route for any traffic that comes through the DHCP LAN IP address to respond via the interface from which the request originated.  Otherwise, forwarded NAT traffic from the main KVM host to the DHCP internal LAN IP will reach its destination, but no response will be sent back (because it will attempt to send the response via the configured static IP address interface which may NOT be the original destination of the senders request).

The Solution:

https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming/23345#answer-23345

From the above link, the solution for me was to do the following in the KVM guest virtual machine:

Only needs to be done once:

sudo -i
echo 200 isp1 >> /etc/iproute2/rt_tables

Setting up the route (adjust variables as necessary):

sudo -i
ip rule add from <interface_IP> table isp1 priority 900
ip rule add from <interface_IP> dev <interface> table isp1
ip route add default via <gateway_IP> dev <interface> table isp

The command I used for my specific setup:

sudo -i
ip rule add from 192.168.122.10 table isp1 priority 900 
ip rule add from 192.168.122.10 dev ens9 table isp1 
ip route add default via 192.168.122.1 dev ens9 table isp1

Making it permanent (apply on system start up):

sudo -i
nano /etc/network/interfaces

I added the below post-up rules (adjust variables as necessary):

auto ens9
iface ens9 inet dhcp
        post-up ip rule add from <interface_IP> table isp1 priority 900
        post-up ip rule add from <interface_IP> dev <interface> table isp1
        post-up ip route add default via <gateway_IP> dev <interface> table isp1

The route is created whenever the dhcp interface is brought up.

Obtaining Let's Encrypt HTTP Validation Server IP Addresses

Use your webserver logs:

sudo apt-get install john
cat access_log.1 | grep "Let's Encrypt" | awk '{print $1}' | unique ips
cat ips

Recreating a Lost or Removed EFI Windows Boot Partition

In case you remove an SSD with a Windows Boot Partition that boots an installation of Windows, you'll need to recreate the boot system to be able to successfully boot again.  To recreate the boot EFI partition on another drive or partition, perform the following:

1) Create or use an existing Windows installation disc or flash drive (you can use Rufus USB to format a flash drive with a Windows ISO)
2) Boot PC using your Windows installation media (a CD, DVD, flash drive, etc containing Windows installation files)
3) Press SHIFT + F10 on the first screen to bring up Command Prompt
4) Run the following commands and click Enter each time at Command Prompt:

diskpart
list disk
select disk N (N refers to the disk which contains the deleted the EFI System partition)
list partition
create partition efi size=200
format quick fs=fat32
list partition
list volume (find the volume letter which belongs to the installed Windows OS)
exit (exit diskpart)
bcdboot M:\windows (M refers to the volume letter of installed Windows OS)

Add any additional Windows installations by repeating the bcdboot command followed by the installation path for any other Windows installations you want added to the menu.  For example, I added my Windows 7 partition using the below command:

bcdboot P:\windows

You should now be able to boot Windows again without needing the old boot partition or drive.

Instructions were modified from this post.

Ryzen Windows 10 Random BSOD Fix

For the longest time, I could not figure out why I couldn't get Windows 10 to consistently install or run in a stable fashion for longer than a few minutes on my Ryzen 1700X or my Ryzen 3900X AMD CPU. 

After trying everything I could think of and pulling my hair out, I found a simple solution that appears to have worked!  Simply disable Link State Power Management in your Power Setting's active power plan.

Go to the "Control Panel" -> "Power Options" -> Click on the "Change plan settings" link for your selected power plan -> click on the "Change advanced power settings" link -> navigate to the "PCI Express" category -> find the "Link State Power Management" option -> set it to "Off" (for both on battery and plugged in).

In general, there are several reasons why a computer can suffer a catastrophic Blue Screen of Death (BSOD) issue, but in my case, it turns out I just needed to adjust some power settings to compensate for a storage controller bug found in some solid state hard drives.  Thanks Reddit users for helping me find the fix!

https://www.reddit.com/r/Amd/comments/cx3rpb/new_system_with_ryzen_and_windows_10_freezes/#t1_eykk63l

Run Everything as an Administrator in Windows 10 by Default – Same Windows 7 Behavior

To run everything as an administrator in Windows 10, there are several settings that must be changed.  In Windows 7, you would only have to disable UAC for the current user:

This isn't good enough in Windows 10.  You have to disable UAC for the current user AND do the following:

Run gpedit.msc to open the Local Group Policy Editor. Expand Computer Configuration, Windows Settings, Security Settings, Local Policies, and Security Options. Four settings need to be updated:

1. Set "User Account Control: Behavior of the elevation prompt for administrators in Admin Approval Mode" to Elevate without prompting.
2. Set "User Account Control: Detect application installations and prompt for elevation" to Disabled.
3. Set "User Account Control: Run all administrators in Admin Approval Mode" to Disabled.
4. Set "User Account Control: Only elevate UIAccess applications that are installed in secure locations" to Disabled.

The LGPE automatically saves all changes, so exit it and reboot.

Please read more here:

https://superuser.com/questions/1002262/run-applications-as-administrator-by-default-in-windows-10

Common Internet File System (CIFS) – Login Using Anonymous (Guest) Account to Network Shares & NAS Systems

Windows 7:

To map and connect to a network share that is using the SMB1 protocol in Windows, there are a few things that you need to do depending on which version of Windows you use.  In Windows 7, it should be pretty easy.  When mapping the network drive, be sure to check the "Connect using different credentials" box.  For the login, use "anonymous".  Leave the password field blank (don't provide a password).

Windows 10:

Windows 10 doesn't support the SMB1 protocol by default.  However, it can be enabled.  To enable SMB1 support, go to the Control Panel, click on "Programs and Features", and then click on the "Turn Windows features on or off" link in the left sidebar.  Under the "SMB 1.0" category, enable the "SMB 1.0/CIFS Client" by clicking the checkbox and making sure it's in a checked state.  Uncheck the "SMB 1.0/CIFS Automatic Removal" entry if it's enabled as it will cause anonymous logins to SMB1.0 shares to fail.

The next step is to configure Windows 10 to allow anonymous logins to network shares.

To enable access under the guest account from your computer, you need to use the Group Policy Editor (gpedit.msc). Go to the section: Computer Configuration -> Administrative templates -> Network -> Lanman Workstation. Find and enable the policy "Enable insecure guest logons". These policy settings determine whether the SMB client will allow the guest logon to the SMB server.

More Detailed Guide | Archived Copy

Windows 7 and 10:

If you get a message that a drive is already mapped using different credentials, simply map the connection using its IP address instead rather than its name.