Security Toolbox
Hardening Linux for Production Use
To protect your production server from attacks, employ these common security tools to help safeguard your system.
Special Thanks: This article was made possible by support from Linux Professional Institute
A standard server or minimal install (CentOS) provides you with a secure server system, but its production usefulness is very limited. To make a server production ready, you’ll have to install software such as web services, DNS, DHCP, Samba/CIFS, print services, and databases. Because these services communicate over the network, security vulnerabilities are an inevitability. Exposing any service provides an attack vector. However, there are actions you can take to protect your systems and your data from attacks.
There are too many utilities available, both free and commercial, to help secure your systems. This article provides you a brief overview of the most common methods and utilities to help you maintain system security. Please remember that security is a best effort and that no system is 100 percent secure, but using these hardening suggestions will make it more difficult for malicious actors to compromise your systems.
Low-Hanging Fruit
At a minimum, you must prevent easy hacks on your system by updating it as soon as it comes online and then at least weekly thereafter. Developers release security and performance updates daily. Systems must be kept up to date, or they are vulnerable to attack. The other minimum barrier to attack is to enforce complex passwords. Don’t allow users to use simple, dictionary passwords on systems. Additionally, users should be forced to change passwords every 90 days and prevented from reusing the same password.
You can also minimize local (insider) hacks by logging off when you leave your system unattended. This rule applies to servers, workstations, and SSH sessions alike. This simple act will prevent any malicious user passing by your unlocked computer from stealing root access.
Don’t minimize the obvious protections such as regular backups and physical security. Performing backups is a security method that many administrators overlook. Destructive malware that might change critical files can be easily restored from a backup to their former states. Limiting physical access to systems is a good deterrent for those who want to steal directly from a system with a USB drive or who want to see if an administrator has remained logged in to a console. Only administrators need access to areas where servers are housed.
Multifactor Authentication
Multifactor authentication (MFA) is the best line of defense against password cracking by guessing, brute force, or dictionary attacks. Adding a second factor to logins is a very strong deterrent for account hacks. However, MFA isn’t 100 percent foolproof as some hackers have devised ways to phish your MFA tokens. Before logging in, you should always be aware of where you are entering your passwords and any codes or tokens for additional factors. MFA deters most low-level hackers. Professional hackers and advanced persistent threat (APT) groups will resort to phishing and other social engineering methods to acquire a token, code, or one-time password (OTP) from you.
The answer to these so-called soft tokens and their vulnerabilities is hardware tokens, such as YubiKeys, which are USB-based hardware tokens that require fingerprint verification as the second factor. These hardware tokens require something that you are (fingerprint) rather than something you know (password, passphrase, or picture recognition) or even something that you have (a hardware token that generates random numbers).
Pruning Unused Services
When it comes to system services, less is more – meaning that you should run as few system services as possible while still maintaining a productive environment for yourself and your users. I’ve always used the “one service per server” rule. Virtual machines make this practice a lot more economical than purchasing one physical system per service. One system per service is a good idea, because, if your Samba/CIFS server is compromised through some exploited vulnerability, your other servers that are not running Samba/CIFS will be protected – that is if you’ve implemented MFA.
You can begin with a minimal CentOS installation, which provides one external service: SSHD. An external service means one that is available to other hosts on the network as a service or listening port. Whichever distribution or installation you choose, you should perform a quick test to check which services are set to automatically start and which ones are listening for network connections.
There are two commands that provide you with the information required for making decisions about which services and protocols to trim and which ones need protection and monitoring. The first is chkconfig (Listing 1), which checks the runlevel information for system services.
Listing 1
chkconfig
$ chkconfig netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The second command, nmap, scans a system for open ports. The nmap command can be used locally or remotely. You should do both. Run an nmap scan locally to determine which services are network-facing and which ones are localhost only. Performing the same scans remotely and locally will help determine if there are any rogue services running on your systems. It will also help you to protect ones that are legitimate but may be overlooked by other methods.
The following nmap commands (Listing 2) work both locally and remotely. For remote scans, use the remote host’s IP address and localhost for local scans.
Listing 2
nmap Scanning for Open TCP Ports
# nmap –sT IP_Address Remote: # nmap -sT 192.168.1.59 Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:08 CDT Nmap scan report for 10.50.48.59 Host is up (0.74s latency). Not shown: 999 filtered ports PORT STATE SERVICE 22/tcp open ssh Nmap done: 1 IP address (1 host up) scanned in 60.18 seconds Local: # nmap -sT localhost Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:07 CDT Nmap scan report for localhost (127.0.0.1) Host is up (0.0016s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 998 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
The nmap command in Listing 3 listsboth UDP and TCP ports for a host. You may replace localhost with the IP address of the remote host you want to scan.
Listing 3
nmap Scanning for Open UDP and TCP Ports
# nmap -sTU localhost Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:25 CDT Nmap scan report for localhost (127.0.0.1) Host is up (0.0014s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 1997 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 68/udp open|filtered dhcpc Nmap done: 1 IP address (1 host up) scanned in 1.42 seconds
The lsof (LiSt Open Files) command (Listing 4) displays a list of processes and to which ports they are bound.
Listing 4
lsof
# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME chronyd 2429 chrony 1u IPv4 20062 0t0 UDP localhost:323 chronyd 2429 chrony 2u IPv6 20063 0t0 UDP localhost:323 dhclient 2766 root 6u IPv4 22028 0t0 UDP *:bootpc sshd 2959 root 3u IPv4 22893 0t0 TCP *:ssh (LISTEN) sshd 2959 root 4u IPv6 22909 0t0 TCP *:ssh (LISTEN) master 3207 root 13u IPv4 23647 0t0 TCP localhost:smtp (LISTEN) master 3207 root 14u IPv6 23648 0t0 TCP localhost:smtp (LISTEN)
The netstat command (Listing 5) lists listening TCP and UDP ports for a system. Those listed as localhost can only be reached on the local system, but those listed as 0.0.0.0:<protocol> are accessible over the network. You can’t eliminate all listening ports, because doing so would render your system useless as a server, but these commands provide you with a snapshot view of potential entry vectors to your system by malicious actors.
Listing 5
netstat
# netstat -l Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN tcp6 0 0 [::]:ssh [::]:* LISTEN tcp6 0 0 localhost:smtp [::]:* LISTEN udp 0 0 0.0.0.0:bootpc 0.0.0.0:* udp 0 0 localhost:323 0.0.0.0:* udp6 0 0 localhost:323 [::]:* raw6 0 0 [::]:ipv6-icmp [::]:* 7
Controlling Root User Access
System administrators know the power of the root user account. And that power is absolute on a Linux system. There is nothing the root user cannot do, including removing or changing every file on the filesystem. But root access doesn’t simply apply to logging into a system as root – in fact that is a privilege that no one should have. Only a console root login should ever be allowed. In other words, no root access across the network via SSH or any other protocols. Root user access is typically controlled by a super secret password that only a few privileged individuals know. However, there is also root access that can be used via the sudo command. sudo is the standard method that system administrators use to issue commands as root but without the responsibility of being the root user.
The sudo command is generally limited to system administrators. It keeps everyone on a system honest in that sudo must be used before each command that normally would be issued by the root user account. For example, only the root user can stop and start network interfaces:
# ifdown eth0 # ifup eth0
If a user has sudo access, she can issue this command as:
$ sudo ifdown eth0 Password:
The sudo privileged user must enter her own password to prove she is indeed the owner of the account she is using and that she has sudo access. The value of sudo is that commands are entered purposefully using the sudo command and only one command at a time. Once the privileged user has issued a sudo command and prompted for her password, the authentication remains for five minutes only, which is another way of keeping everyone honest by having privileged users reauthenticate periodically.
Alternatively, if a system administrator must become the root user to perform a series of tasks that requires the root user’s environment, she can become root by using su -:
$ su – Password:
This time the password that’s required is not the user’s password but the actual root user’s password. The advantage of becoming root is that the user then is root and has root’s environment, which is not available to users who use the sudo command. It is imperative that the actual root user’s password be kept secret and only shared with those who really require it. It should also be changed often, which translates to every 30 days for standard production environments.
If administrators have more than a handful of systems to manage, this process can be a bit tedious. The use of LDAP, Active Directory, or NIS+ is recommended in these cases to decrease the number of systems on which the root user’s password must be changed. Having a centralized management interface can also prevent mistyped passwords during a password change event.
Preventing Service Account Logins
Accounts that don’t require shell access should have no standard shell access in the /etc/passwd file. Only actual interactive users and the root user should have a real shell, such as /bin/bash, /bin/ksh, or /bin/csh, for example. All other accounts should either have /bin/false or /bin/nologin for their shell designations, which means that they cannot log in to the system interactively.
Certain accounts will have “pseudo” shells, such as shutdown, sync, and halt, which when used perform those functions. They’re still around for legacy reasons, which were used to allow non-root users the ability to halt or shutdown a system. Unix tradition is that you issue the sync command twice before reboot or halt (shutdown) to write unwritten data to disk. This isn’t necessary anymore, because the system does this itself prior to reboot or halt.
Denying Network Access to the Root Account
It’s easy enough to fix this one, because root logins are disabled by default. Some system administrators enable this feature, including myself, but it’s generally not a good idea. The one possible exception to this is if you have a hardware token and multifactor authentication setup in your environment. But for 99.99 percent of us, leaving this option disabled is exactly what you want to maintain a higher level of security.
The reason for denying network access to the root account might not be immediately obvious. Since most accounts are only protected by a password, root account compromise is only a guess or a brute force attack away. Because of this, you should make becoming root as difficult as possible without impeding productivity.
Auditing Security
There are many applications and application suites available to perform regular security audits and track changes to critical files. In addition, you have the power to perform a quick security audit on your system by using the find command. This is by no means a deep security audit but just a quick and dirty scan of your system to find files that have the SUID/SGID permission set that shouldn’t. Having these permissions bits set on files owned by root is a serious vulnerability, because they can allow users who do not have root access to execute commands as root and potentially initiate a root account compromise.
What you’re looking for is files that have permissions set similar to the following:
-rwSr-xr-x (SUID) or –rw-r-Sr-x (SGID)
The following find command locates all files with the SUID permission set for root:
# find / -type f -perm -u=s -ls
Here are some examples of files that match this permission:
12734848 44 -rwsr-xr-x 1 root root 44320 Mar 14 05:37 /usr/bin/mount 12734863 32 -rwsr-xr-x 1 root root 32208 Mar 14 05:37 /usr/bin/su 12734867 32 -rwsr-xr-x 1 root root 32048 Mar 14 05:37 /usr/bin/umount 13025626 144 ---s--x--x 1 root root 147392 Oct 30 2018 /usr/bin/sudo 12777317 60 -rwsr-xr-x 1 root root 57664 Nov 20 2018 /usr/bin/crontab 13037754 28 -rwsr-xr-x 1 root root 27832 Jun 10 2014 /usr/bin/passwd
Similarly, the find command to locate files that have the SGID permission set is as follows:
# find / -type f -perm -u=s -ls
And here are some examples of matching files:
12649555 16 -r-xr-sr-x 1 root tty 15344 Jun 9 2014 /usr/bin/wall 12734873 20 -rwxr-sr-x 1 root tty 19624 Mar 14 05:37 /usr/bin/write 13015059 376 ---x--s--x 1 root nobody 382240 Apr 10 2018 /usr/bin/ssh-agent
As you can see from you own listing, the number of files with SGID set is far fewer than those with SUID set. Some system files require SUID/SGID permission to be set, but there are very few of them. You need to perform a baseline audit of your systems upon initial installation and then track those changes periodically to be sure that no rogue programs or users have exploited this security flaw.
Conclusion
There are several other Linux hardening methods such as PAM, iptables, enabling SELinux, removing any X display managers, and regular port monitoring, but these are outside the scope of this article. I may revisit them in future installments individually. As stated previously, you can’t remove all network access to your servers, because that defeats the purpose of having a server. But now you have a small but powerful toolbox of utilities and techniques that will help you to keep your systems safer.