The Dangers of Using tcp_tw_recycle in Linux – Strange Intermittent Timeout Issues

Wednesday, January 13th, 2021

Do Not Use tcp_tw_recycle

I had a very strange connectivity issue recently that I was only able to reproduce intermittently on my own LAN network when connecting to a few of my servers hosting websites that process and receive tons of simultaneous connections at any point in time.  Basically, my connection to a specific set of websites that I host would timeout from my home internet connection.  However, I was never able to reproduce this issue when connecting to the same sites from other networks belonging to my family and friends. 

From my home connection, I used TCPView and saw that SYN_SENT packets were supposedly sent to my servers to establish a connection.  Unfortunately, the server never replied to some of these requests.  As such, my connection would timeout at times, and work perfectly fine sometimes.  I looked at DD-WRT's connection table, and it also claimed that the packets had been sent, but that they were in an UNREPLIED state when I experienced issues.  Thus, packets were supposedly being sent, but the server was not responding at times.  After spending nearly a week trying to tackle this issue and buying new cable internet equipment (an officially supported Comcast modem), I tracked down the issue, and it ended up being a TCP configuration setting on my servers rather than my home LAN equipment.

Modem or Router's Fault?

Originally, I thought my issue was caused by the DD-WRT open source firmware I was running on my wireless router.  If I restored the router's settings to DD-WRT's factory defaults, I could always connect to the websites I was having intermittent connection timeout issues on.  I suspected it might be my router after trying an older router which didn't have any problems either.  I even upgraded the DD-WRT firmware to the latest version and rebuilt my complicated network configuration from scratch.  Unfortunately, the issue was still there.  Thus, despite mixed results with different routers, I started to wonder if the issue was on my server's end.

Finally Fixed

I started looking at sysctl TCP settings I could adjust on my router, and I ended up comparing some of these values to the ones used on my servers (that were hosting the problem websites).  Eventually, I came across configuration values I had changed myself several months ago which were supposed to help the server support multiple simultaneous connections.

After reading this StackOverflow thread (https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both), I decided I would try disabling the tcp_tw_recycle setting.

/proc/sys/net/ipv4/tcp_tw_recycle was set to 1 (enabled) from tweaks I had run that I had found on the internet.  After I disabled it, /proc/sys/net/ipv4/tcp_tw_recycle was set to 0 (disabled).  By default, Linux keeps tcp_tw_recycle disabled.  Again, this is something I had changed for tuning reasons.  After disabling this setting and rebooting the server, I no longer have any issues connecting to the severs in question.  No more connection timeouts, and everything works properly again.

I have no idea why I wasn't able to reproduce this issue on other networks.  I thought it was my network equipment (modem and router), but it ended up being the server.

Lessons Learned

Be careful when applying settings you find online.  Sometimes, they may not work, or their usage may be buggy.  In fact, net.ipv4.tcp_tw_recycle has been removed from Linux in kernel versions newer than 4.12 by default.  I'm guessing this is because it doesn't work, as I experienced.  Do NOT use  net.ipv4.tcp_tw_recycle! I kept tcp_tw_reuse enabled, so you can enable this setting without running into problems.  Just don't for the love of anything use tcp_tw_recycle!  It doesn't work, and it will cause you headaches trying to track down strange intermittent issues!

 

Linux Multiple Network Interfaces (NICs) – One Interface with Static Public IP and One Interface with Private DHCP LAN IP Address – Routes and Routing

Friday, July 24th, 2020

Linux KVM:  Using Multiple NICs and Routing Traffic Properly Between Them

When setting up a KVM guest to use multiple network interface controllers (NICs), additional ip routes may be needed in order for the additional interfaces to work properly.  For example, if you configure a NIC with a public static IP address and a NIC with an internal private DHCP LAN IP address, you must create a route for any traffic that comes through the DHCP LAN IP address to respond via the interface from which the request originated.  Otherwise, forwarded NAT traffic from the main KVM host to the DHCP internal LAN IP will reach its destination, but no response will be sent back (because it will attempt to send the response via the configured static IP address interface which may NOT be the original destination of the senders request).

The Solution:

https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming/23345#answer-23345

From the above link, the solution for me was to do the following in the KVM guest virtual machine:

Only needs to be done once:

sudo -i
echo 200 isp1 >> /etc/iproute2/rt_tables

Setting up the route (adjust variables as necessary):

sudo -i
ip rule add from <interface_IP> table isp1 priority 900
ip rule add from <interface_IP> dev <interface> table isp1
ip route add default via <gateway_IP> dev <interface> table isp

The command I used for my specific setup:

sudo -i
ip rule add from 192.168.122.10 table isp1 priority 900 
ip rule add from 192.168.122.10 dev ens9 table isp1 
ip route add default via 192.168.122.1 dev ens9 table isp1

Making it permanent (apply on system start up):

sudo -i
nano /etc/network/interfaces

I added the below post-up rules (adjust variables as necessary):

auto ens9
iface ens9 inet dhcp
        post-up ip rule add from <interface_IP> table isp1 priority 900
        post-up ip rule add from <interface_IP> dev <interface> table isp1
        post-up ip route add default via <gateway_IP> dev <interface> table isp1

The route is created whenever the dhcp interface is brought up.

Installing Chrome WebDriver (Linux Script)

Wednesday, August 28th, 2019

Installing Chrome WebDriver (Linux Script)

Find out which version of Chrome is installed on your system before running the below commands.  You can find out your chrome version by running the following command:

google-chrome --version

Adjust the version number (replace {VERSION_NUMBER})  in the below commands to match the version installed on your system!!!

sudo -i
cd ~/Downloads
rm chromedriver_linux64.zip
wget -N https://chromedriver.storage.googleapis.com/{VERSION_NUMBER}/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver

Selenium and other libraries that rely on the Chrome WebDriver should now work properly.

Change the Default Editor to nano in Linux

Saturday, April 27th, 2019

Use nano as the Default Editor

If you hate vi like I do, you can configure Linux to always default to using the nano editor.

Simply add the following to the bottom of the /etc/bashrc file:

export EDITOR="nano"

Save the file.  nano is now the default editor.  When you use

sudo crontab -e

The nano editor will now be used by default.

Ubuntu Grub Fails to Install on RAID Array

Friday, February 6th, 2015

Ubuntu Grub RAID Issues

Grub Fails To Install on RAID Array

If grub fails to install on your RAID array in any version of Ubuntu, do NOT disable your BIOS RAID! The correct solution is at this blog entry. I'll summarize it below.

At the stage of the install where it is attempting to install GRUB it will detect as

/dev/mapper

This is incomplete! That's why the GRUB install fails.

You need the actual name of the RAID array to install to. So during that step, press ctrl+alt+F2 to drop to a busybox terminal, then enter

ls -l /dev/mapper

Pick out the name of your array from the list shown, then press ctrl+alt+F1 to switch back to the install (you can switch back and forth as much as you like with no problems) and enter it in the field as

/dev/mapper/{your array name}  

Then GRUB installs perfectly and you're ready to go, with a proper BIOS RAID array intact.

System Won't Boot After Grub Failed to Install

If your system will no longer boot because you skipped installing or updating grub, you need to download an Ubuntu version that does support RAID, boot from the LIVE CD, drop to a terminal, and then run:

ls -l /dev/mapper
sudo grub-install /dev/mapper/{ARRAY_NAME_HERE}

Setting Up RAID Array During Ubuntu Install

If you are configuring a BIOS RAID array for the first time on Ubuntu, you should create a 1MB boot partition.  Its partition type is "boot".  If you do this, grub will always try to install there and will succeed every time without failing when upgrading or reinstalling grub.

Install VSFTPD 3.0.2 on the ARM Platform for Raspbian or Debian

Saturday, January 3rd, 2015

Install VSFTPD 3.0.2 on the ARM Platform for Raspbian or Debian

Download and install the arm compiled VSFTPD 3.0.2 deb package file and init script:

wget -N http://dinofly.com/files/linux/vsftpd_3.0.2_arm.tar.gz
tar -zxvf vsftpd_3.0.2_arm.tar.gz
if [ ! -e "/etc/init.d/vsftpd" ]; then
    sudo cp vsftpd /etc/init.d/
fi
sudo dpkg -i vsftpd_3.0.2-1_armhf.deb

Now, run these commands:

sudo useradd -d /var/ftp ftp
sudo mkdir -p /var/ftp
sudo chown root.root /var/ftp

Finally, start the VSFTPD service and set it to run on boot:

sudo service vsftpd restart
sudo update-rc.d vsftpd defaults

VSFTPD 3.0.2 should now be installed on your ARM device.

Geany PHP Beautifier

Saturday, August 16th, 2014

PHP Beautifier Support for Geany

Geany is by far one of the best text editors I have come across that works on both Windows and Linux.  It is also one of the most aesthetically pleasing editors to look at right out of the box.  I do a lot of PHP scripting, and as such, it is nice to have a "beautifier" script that will automatically format my code for me so that it looks nice.  Geany can also call the php executable and check your script syntax. You can achieve both of these features by installing both PHP for Windows and the PHP Beautifier PEAR addon.   

Install PHP for Windows:

If you code your PHP scripts in Windows, you'll want to use syntax checking and the PHP_Beautifier script.  To do so, you must install the PHP5 Windows package, which includes the main PHP binaries.

To get the files, download the latest version of PHP 5.3.

Extract the contents of the archive to "C:\php5"

Go into C:\php5 and rename "php.ini-development" to "php.ini".

PHP BEAUTIFIER WILL NOT WORK ON ANY NEW VERSION OF PHP FROM 5.4.X and UP!

Install PEAR for Windows:

PHP_Beautifier relies on PEAR functionality.  To install pear, save this file using a browser and place it in the "C:\php5" directory.  

Start command prompt, change directory into "C:\php5", and run the phar script:

C:\
cd C:\php5
php go-pear.phar

Install everything and keep default options.

Install PHP_Beautifier:

Now, install PHP Beautifier by running the following commands:

pear install PHP_Beautifier

Integration in Geany:

For PHP Syntax Checking:

In Geany, click on "Edit" in the menu bar and choose "Preferences".

In the "General" and "Startup" tabs, under the "Paths" section, paste "C:\php5" (without the quotes) into the "Extra plugin path:" field.

For PHP Beautifier:

Start the Geany text editor program.  Open a PHP script file.

Select the code you want to format, right click on the selected text, and choose "Format" –> "Send Selection to" –> "Set Custom Commands".

For command, use the following:

php C:\\php5\\php_beautifier -s4 -l "ArrayNested() NewLines(before=T_COMMENT:for:switch:foreach:T_CLASS:function:T_CLOSE_TAG,after=T_ENDIF:T_CLOSE_TAG:T_OPEN_TAG:T_ENDSWITCH:T_ENDWHILE:T_ENDFOR:T_ENDFOREACH)"

For "Label", use "PHP Beautifier"

Hit OK.

Now, select the code you want to format, right click on the selected text, and choose "Format" –> "Send Selection to" –> and pick "PHP Beautifier".  The code should now be formatted using the options specified in the command line arguments above.

For more filter options and commandline parameters, please read this PHP_Beautifier document.

 

Turn on IPv4 Easy Bash Way

Tuesday, January 21st, 2014

Turn on IPv4 Forwarding by running this script:

cd ~/Downloads
wget -N "http://dinofly.com/files/linux/ipv4_forward.tar.gz"
tar -zxvf ipv4_forward.tar.gz
sudo bash forwarding.sh

It should work on all versions of Linux but has been tested and works perfectly on Ubuntu.

How to Make MATE Look Like Windows XP using the Luna Theme

Sunday, April 14th, 2013

Make MATE or GNOME2 Look Like Windows XP Using the Luna Theme

If you want your Linux installation to look like the original theme used in Windows XP, you can do that! This guide will walk you through the process of easily making any MATE or GNOME2 Desktop Environment look like the Windows XP GUI. The Luna Theme can be downloaded here and installed using our simple installation script. If you already have MATE installed or are already running GNOME2, skip to the Luna Theme install instructions.

Install MATE on Ubuntu:

Run the below commands for your matching Ubuntu version in a terminal to install MATE.  To find out which version of Ubuntu you're running, use this command:

lsb_release -a

For Ubuntu 12.04:

sudo add-apt-repository "deb http://packages.mate-desktop.org/repo/ubuntu precise main"
sudo apt-get update 
sudo apt-get --yes --quiet --allow-unauthenticated install mate-archive-keyring 
sudo apt-get update
sudo apt-get install -y mate-desktop-environment

For Ubuntu 14.04:

sudo apt-add-repository ppa:ubuntu-mate-dev/ppa
sudo apt-add-repository ppa:ubuntu-mate-dev/trusty-mate
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y mate-desktop-environment-extras

For Ubuntu 16.04:

sudo add-apt-repository ppa:ubuntu-mate-dev/xenial-mate
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y mate-desktop-environment

For Other Distributions (Distros):

http://wiki.mate-desktop.org/download

Installing the Luna Theme:

Our version of the Luna theme has been converted and ported over to GTK3, so it should work with all newer flavors of Linux running MATE while still working on older Linux installs running GNOME2.To install the Luna Theme which will make Linux look like Windows XP, run the following commands. The theme files will be downloaded and saved in your Downloads directory.

cd ~/Downloads
wget -O linux_xp_luna_theme_install.tar.gz http://dinofly.com/files/linux_xp_luna_theme_install.tar.gz
mkdir Luna
tar -zxvf linux_xp_luna_theme_install.tar.gz -C Luna
cd Luna
sudo rm -rf /usr/share/themes/Luna
rm -rf ~/.themes/Luna
sudo bash install.sh

Next, Right Click on the Desktop, and choose "Change Desktop Background".  Click on the "Themes" tab.  Select "Luna".  Click on the "Background" tab.  If you want the default XP wallpaper set as your background, click on the "Add" button.   Select your "Pictures" folder.  Select "luna_background.jpg".  Click "Open".  Click on "Close" to change it. 

Now, MATE or GNOME2 looks like XP!  Enjoy!  This theme was copied from Ylmf OS 3.0.

Problems Compiling the Ralink Wireless N Driver Make Error 2 Fix

Tuesday, January 22nd, 2013

Problems Compiling the Ralink Wireless N Driver

If you receive a make error 2 while compiling the ralink drivers, try this fix. This applies to newer kernel versions.