Detecting RegreSSHion with Dynatrace (CVE 2024 6387)

The Qualys Threat Research Unit (TRU) has discovered a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The CVE assigned to this vulnerability is CVE-2024-6387.

The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, which is a significant security risk. This race condition affects sshd in its default configuration.

Potential Impact of regreSSHion

Exploiting this vulnerability can lead to full system compromise, where an attacker can execute any commands or code with the highest privileges. This can result in a complete system takeover, malware installation, data manipulation, and the creation of backdoors for persistent access.

Moreover, gaining root access will enable attackers to bypass critical security mechanisms such as firewalls, intrusion detection systems, and logging mechanisms, further obscuring their activities. This can result in significant data breaches and leakage, giving attackers access to all data stored on the system, including sensitive or proprietary information that can be stolen or publicly disclosed.

To avoid sounding too alarming, we have to emphasize that this vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. As written in the Qualys white paper, "In our experiments, it takes ~10,000 tries on average to win this race condition. " To our knowledge as of writing this article, no properly working exploits have been created.

How to mitigate the risk

The following versions of OpenSSH are impacted by this vulnerability:

• Versions earlier than 4.4p1 (unless patched for CVE-2006-5051 and CVE-2008-4109)
• Versions from 8.5p1 up to, but not including, 9.8p1

OpenBSD systems are unaffected by this, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

There are three strategic paths to take to mitigate this threat:

• Patch Management: Quickly apply available patches for OpenSSH or upgrade to the relevant version that is not impacted by this vulnerability;
• Access Control: Limit SSH access via network-based restrictions;
• Network Segmentation: Divide networks to restrict unauthorized access and lateral movements from other systems.

Detect exploitation attempts with Dynatrace

This vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. Thanks to this noisy approach it can be detected from your sshd logs with Dynatrace DQL queries using Dynatrace Security Investigator.

Look for timeout events

Exploitation attempts for this vulnerability can be identified by many lines of "Timeout before authentication" in the logs. The log line in the logs could look something like this:

Jul  1 09:40:39 [localhost] sshd[787404]: fatal: Timeout before authentication for 192.168.12.45 port 17296

To search for such events from your sshd authentication logs stored in Grail, the following DQL query can be used:

fetch logs, from: -48h
| filter contains(content, "Timeout before authentication")

If you don't get any results, that might be good news! You can further expand the timeframe of your query by changing the from time to -7d. This will get you logs from last week to make sure that no one tried to exploit the system during the last week.

Look for a high volume of log entries

There's a chance that someone has tried to attack your system with a different attack or that the attack did not succeed as intended. For example, some Proof-of-concept attacks have failed, and these failures write various error messages to the victims' sshd logs. It's always good to check the log trend line in general to detect such attacks and when hunting for the unknown.

It's easy to create a timeseries from your logs using DQL. With the following query, you can create a chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval.

fetch logs, from: -24h
| filter contains(content, "sshd[")
| makeTimeseries count(), by: { dt.entity.host }, interval:1h

From the chart, it's easy to see that nothing anomalous was detected using this query: no weird peaks or anomalies could be extracted from the chart.

Look for suspicious IP addresses in sshd logs

To ensure no one has tried to access your services from a suspicious location, it's valuable to extract all the IP addresses from your sshd logs and see if something suspicious emerges from there. To do that, the following DQL query is helpful:

fetch logs, from: -2d
| filter contains(content, "sshd[")
| parse content, "ARRAY{ ld* ipaddr:ip }{0,10}:source_ip"
| expand source_ip
| summarize count(), by:source_ip
| sort source_ip desc

This query will fetch your sshd logs from the past two days, will extract ALL the IP addresses (both ipv4 and ipv6) from the content and will sort the result based on their occurrances to your results. If you discover any suspicious IP addresses from this list, you can start your investigation based on the discovered threat! If you see only your own addresses, you can rest assured: your sshd has not been accessed by an unknown IP address.

Analyze network flow logs

Last but not least, your network logs are the ultimate source of data. If you're using AWS for your cloud platform and you have forwarded your VPC flow logs to Dynatrace, you can use DQL to look for your potential victims AND suspicious IP addresses that have tried to access your servers using SSH.

Consider the following query:

fetch logs, from: -24h
| filter aws.log_group == "/aws/vpc/flow_logs"
| parse content, """INT:version ' '
NSPACE:account_id ' '
NSPACE:interface_id ' '
('-' | IPADDR:src_addr) ' '
('-' | IPADDR:dst_addr) ' '
('-' | INT:src_port) ' '
('-' | INT:dst_port) ' '
('-' | INT:protocol) ' '
('-' | LONG:packets) ' '
('-' | LONG:bytes) ' '
TIMESTAMP('s'):start ' '
TIMESTAMP('s'):end ' '
LD:action ' '
LD:log_status"""
| filter dst_port == 22

The query will fetch all the Grail logs ingested from the CloudWatch log group called /aws/vpc/flow_logs. Using the VPC flow log default pattern available in DPL Architect, we can extract the meaningful fields to see only the network traffic targeting the SSH port.

From this dataset, we can create metrics based on the destination IP address to detect all the potential victims by adding the following code snippet to the previous query:

| filter action == "ACCEPT"
| makeTimeseries count(), by: {dst_addr}, interval: 1min

Viewing the results as a graph, you can see that some servers have received an anomalous amount of SSH traffic towards them! From here, you can continue to look further into the specific destinations to understand if this is malicious activity.

To understand who is targeting you, a similar query can be used. Instead of dst_addr use a src_addr to see the most active sources that have been targeting any of your ssh daemons on all the servers in the respective VPC.

| filter action == "ACCEPT"
| makeTimeseries count(), by: {src_addr}, interval: 1min

The results show that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis.

Next steps

This particular vulnerability is challenging to exploit due to its race condition nature. However, all vulnerabilities should be taken seriously, their potential impact on your environment analyzed, and, where possible, mitigated. If you plan not to take any action for whatever reason, you should at least analyze the potential risk and make an informed decision before taking any action.

Public link

Please use the above public link if you want to share this noodl on another website.