A study in detecting network intruders
Uninvited Guests
The nightmare of any admin is a user who can't resisting clicking on an unknown attachment labeled Application.exe. This article draws on a real-world example to show how you can use built-in Linux resources to detect unauthorized traffic that might have been invited in by a trigger-happy user.
If a network monitoring process detects malware, a system administrator needs to identify the affected systems and contain the damage. A customer from a Microsoft-heavy environment recently came to me with a problem. Many of the clients on his network were infected with malware. Because the malware was quite sophisticated, the virus scanner did not help detect it. He wanted my help with finding all the infected clients.
To make things even more exciting, several versions of the malware appeared on the network. The variations in form meant that it was not easy to detect the malware using simple pattern matching with a filesystem scan. Fortunately, although the attackers were good at infiltration, they were not very skilled at connecting back to their Command and Control (C&C) server. This article describes our investigation and offers some tips on how to respond to similar attacks.
The Malware
The first generation of the malware attempted to connect to Telnet servers in Asia via TCP port 23. Since the Telnet protocol is hardly ever used on today's networks, this attack was quickly noticed. A single tcpdump
[1] command directed at the gateway returned some initial hits:
tcpdump -i eth0 tcp port 23
The packets had some peculiarities in the fields of the Telnet header, which were revealed by an analysis with Wireshark [2]. Headers are typically used to negotiate things like terminal emulation or even X forwarding. In the case of the malware, the headers contained nonsense, but we always found the same 4-byte sequence at the beginning of the packet.
The second generation of the malware tried to disguise itself as a DNS packet. This time it used UDP port 53, but it wanted to contact servers on the same Asian network. The desired DNS traffic on the network moved back and forth between clients and the Active Directory domain controller, because the domain controller machine was acting as a DNS server. Undesirable traffic was filtered out using tcpdump
:
tcpdump -i eth0 port 53 and not host IP_address_of_AD_controller
tcpdump
detected the traffic that went to port 53 but not to the Active Directory server. The analysis with Wireshark also showed that the sequence already known from the Telnet packets reappeared at the beginning of the packets. In addition, the packets were again not valid DNS packets.
The last version of the malware finally tried HTTP port 80, but it was caught due to invalid HTTP packets. In addition, starting in version 2 of the malware, DNS queries had already been noted against different hosts in three different subdomains. The victim of the attack had several sites, so there were several Internet gateways.
The Search
Although most of the network was Microsoft, the gateways and DNS servers fortunately ran on Linux. I was thus quickly able to launch a search for clues.
Many companies allow their employees outgoing access to the Internet via the company gateway and do not log the permitted traffic at the firewall. This approach has some disadvantages: If the logs had been run through a log management solution such as ELK [3] or Splunk [4], it would have been very easy to narrow down the computers because traffic via ports 23 or 53 is fairly atypical.
If web access is first routed through an internal enterprise proxy, computers that do not use this proxy but look for a direct connection to the outside through ports 80 (for HTTP) or 443 (for HTTPS) are at least suspicious.
If the firewall also blocks outgoing connections that do not originate from computers with known services (proxy, DNS server, mail server, and so on), the analyst can simply search for rejected packets that use an address outside the destination. A perfect world should have hardly any packets of this type.
Clients that do send these packets are either configured incorrectly (depending on destination and port) or should be inspected more closely by the admin.
Wireshark: The Forensic Scientist's Friend
Wireshark [2] is the industry's standard tool for investigating network incidents from logged packets. The large number of protocol modules available with Wireshark – for application protocols such as HTTP or SMTP – allows for more in-depth analysis.
At the same time, Wireshark massively supports the admin in finding a needle in a haystack. The software makes it possible to set filters for each protocol field. Figure 1 shows a Wireshark display filter for all HTTP packets that use the POST
method and go to the target host 192.168.1.1
. Wireshark also offers statistical functions that can help with your analysis.
Security experts advise against operating Wireshark with root privileges. Previous examples of errors in software's packet analysis modules have allowed attackers to inject malicious packets into the traffic. See the box entitled "Avoiding Root" for more on running Wireshark without root privileges.
Avoiding Root
The best way to make use of Wireshark without assigning it root privilege is to first create a .pcap
file with tcpdump
and then evaluate it with Wireshark using an unprivileged user ID or a named pipe. Create a pipe with the command:
mkfifo /tmp/wirepipe
Then call Wireshark using:
wireshark -k -i /tmp/wirepipe
Now you can now feed packets into this pipe using the root account and evaluate them using non-root Wireshark. Use the following command, to which you can add a filter for packet types if needed:
tcpdump -w /tmp/wirepipe
Alternatively, you can use the dumpcap
command:
dumpcap -w /tmp/wirepipe
Dumpcap [5] understands some options. For instance, you can specify the interface from which the packets should come. The -f
option lets you specify a filter in tcpdump
syntax.
The Conversations menu entry is the first port of call when searching for infected systems. You may also be able to use this option to find suspicious endpoints outside your own network. If you know the IP address of a control server, in the ideal case, you can quickly find all systems that communicate with it.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.