Best Laid Plans

In recent years, multifactor authentication (MFA) has been a hot topic in information security, with many organizations and software services now making it a requirement. To achieve MFA, two or more authentication factors must be provided by a user to pass authentication. These factors include something you have, something you know, something you are, somewhere you are, or something you do.

Many organizations have turned to the Google Authenticator tool to implement MFA using a time-based one-time password (TOTP). Using TOTP with Google Authenticator satisfies the "something you have" authentication factor because TOTP requires a device in the user's possession (e.g., the user's Android smartphone or iPhone.) Adding a regular user password to satisfy the "something you know" authentication factor provides the second factor to achieve MFA. Many software as a service (SaaS) providers, such as GitHub, AWS, and Microsoft Azure, support Google Authenticator as an option for MFA.

At a high level, TOTP works by having a secret key that is generated on a service and shared with a device. The TOTP algorithm with two inputs, the secret key plus the system's Unix time, results in a one-time password known by both the device and the service. A new password is typically generated every 30 or 60 seconds.

Google provides a pluggable authentication module (PAM), google-authenticator-libpam [1], that system administrators can use to integrate various Linux services with Google Authenticator. As a PAM module, it can be used with virtually any Linux service with robust industry-standard authentication methods. In this article, I will specifically integrate Google Authenticator with SSH logins.

Advantages of MFA with SSH

You will find MFA useful on servers that have SSH open to the entire Internet or have SSH open to large networks. By adding MFA to SSH, you can mitigate brute-force attacks on SSH servers, as well as lower the impact of user password leaks. An attacker would need both the user's password and access to the user's Android or iPhone device (or their Google Authenticator secret key) to gain access to their user account on a server via SSH.

Typically, SSH brute-force attacks on the open Internet are dictionary password attacks, and adding TOTP will negate this type of attack. However, if a password leak is suspected, or a user password seems to have been discovered by an attacker, it is necessary to change the password regardless of whether MFA has been implemented or not.

Considerations

While MFA does provide additional security benefits for authentication with SSH, it does not make sense for all use cases. Bastion hosts, which are typically accessed manually by users or as a jump host, are usually good use cases for MFA. However, MFA may not be ideal for internal SSH hosts located behind a bastion jump host, because two separate MFA codes will be required for both the bastion jump host and the destination server. This could cause some confusion for users on login. If a bastion host configured with MFA is the only SSH server on the network with Internet access, then SSH hosts that are only accessible internally could be considered sufficiently secure with normal password or key authentication.

Large server environments primarily managed with Ansible instead of manual SSH user access pose another problem to using MFA. Ansible relies on SSH to manage configurations on servers. Having separate time-based passwords for each host SSH login during an Ansible playbook run could be cumbersome, even unfeasible, for large server environments.

Installation

The Google Authenticator module requires libqrencode as a dependency for generating QR codes in a shell session. On Ubuntu Server 22.04, libqrencode can be installed with:

# apt install libpam-google-authenticatorlibqrencode4

Time Synchronization

TOTP requires that the system time be accurate, so synching the server with a Network Time Protocol (NTP) service is recommended. While not a requirement for TOTP to work, NTP will ensure system time synchronization with an external service. By default, Ubuntu Server 22.04 uses systemd-timesyncd for system time synchronization, but it does not have an NTP server configured by default. To determine if the NTP service is active, use the following command:

$ timedatectl status

I recommend adding NIST's NTP service as the primary NTP server, and Ubuntu's own NTP service as a fallback. To do this, uncomment the #NTP= and #FallBackNTP=ntp.ubuntu.com lines in /etc/systemd/timesyncd.conf, and change NTP= to NTP=time.nist.gov. After these configuration changes have been made, restart the timesyncd service with:

# restart systemd-timesyncd.service

The system should now synchronize its time with the NIST NTP service.

Configuration

The OpenSSH server must be configured to use PAM. In /etc/ssh/sshd_config, look for the configuration item UsePAM yes. On Ubuntu Server, sshd is configured to use PAM by default. Next, switch the KbdInteractiveAuthentication option from No to Yes.

Once these configuration changes have been added to sshd_config, you will need to add the Google Authentication module to PAM.

PAM looks for service-specific configuration files within /etc/pam.d/ by default. OpenSSH server has a file located at /etc/pam.d/sshd. Add this configuration item to the bottom of line:

auth required pam_google_authenticator.so nullok

The temporary nullok option, used for testing, allows users that have not generated a secret key to still authenticate with SSH using only their password. Once keys have been added to all SSH users, the nullok option will be removed to enforce MFA.

Finally, reload the SSH server configuration with:

# systemctl reload sshd

Generating a Key

Next, you will need to generate a key for users. If you are logging in with the user account via SSH, you will see Verification code: after entering the password or authenticating with your SSH key. Just click Enter. Because you added nullok during PAM configuration, you will still be authenticated. Once logged in, run google-authenticator to generate a secret key. When you run the command, you will be prompted to choose whether the token is time based; enter y for yes.

A QR code will be displayed, which you can scan with the Google Authenticator app on your phone. A plaintext key will also be displayed, which should be treated with the security of a password. You should write down and securely store the plaintext key or save it to a password manager. This plaintext key can be used to re-add the TOTP key to Google Authenticator if the existing phone is lost, replaced, or broken. After adding the QR code or key to your Google Authenticator app, you can enter the current code displayed in the app for the next prompt in the shell console to verify it is working (you can use -1 to skip this, but it is not recommended). You will also receive several "scratch codes," which can be used in emergencies if the Google Authenticator device is not available. Next, you'll be given several prompts for security options; choose y for yes for all of them (see Figure 1) The rate limiting options are especially important because these options will prevent an attacker from attempting a brute-force attack on a time-based password.

Figure 1: For each security question prompt, enter y.

Finalizing

Once a secret key has been generated for every user that will need to use SSH, you must remove the nullok option to enforce MFA. As a critical step before continuing, you must test the google-authenticator module to make sure it is working on your system, as well as confirm that all existing users have set up secret keys and Google Authenticator on their devices. Once this step is complete, in /etc/pam.d/sshd, change:

auth required pam_google_authenticator.so nullok

to:

auth required pam_google_authenticator.so

Then reload sshd:

# systemctl reload sshd

Adding New Users

If you periodically add new users to the server that require SSH, these users will need to generate a key via a direct console login. If a console login is not feasible (e.g., servers running in a cloud service), then a temporary key will need to be generated by the system administrator or ideally by a user creation script. An administrator can generate a temporary key for a new user with the code in Listing 1.

# su - new_user
$ google-authenticator
$ exit

Once generated, you then share the temporary secret key, QR code, or plaintext key with the user. The new user will ideally need to rerun key generation on their first login, which you can enforce with various methods such as using a flag file for new users.

Flag File Enforcement

The /etc/skel directory contains default shell profiles and configurations that are copied into users' home directories on account creation. You will need to update /etc/skel/.profile to check for a flag file. Flag files are empty files that are often used in shell scripts to determine how the script should behave. Append the if block in Listing 2 to /etc/skel/.profile.

# Run google-authenticator if a flag file exists.
if [ -f $HOME/.first_login ]
then
  google-authenticator
  rm -f $HOME/.first_login
fi

The modification from Listing 2 will check if the flag file $HOME/.first_login exists. If so, it will run google-authenticator and then delete the flag file. When new users are added to the server by the administrator, the administrator can manually create the flag file in the new user's home directory with the commands in Listing 3.

# useradd -m new_user
# passwd new_user
# su - new_user
$ google-authenticator
$ exit
# touch /home/new_user/.first_login

To optionally automate this, you could create an empty flag file inside of /etc/skel:

# touch /etc/skel/.first_login

By having the flag file inside of /etc/skel, the flag file will be placed in the new user's home directory automatically, so the administrator will not need to touch the flag file after creating the account.

Conclusion

With simple setup and low administrative overhead, adding Google Authentication TOTP as an authentication method for SSH provides the additional security of MFA for remote server access at a low time cost.

The Author

Jesse Hagewood is a Certified Information Systems Security Professional (CISSP) and an AWS Certified DevOps Professional. He has been a Linux administrator since the start of his career, beginning with managing self-hosted Linux email and web servers for local newspapers and later on administering large SaaS platforms. Currently, he is a DevOps Engineer at Ozmo, Inc. in Blacksburg, Virginia