The Case for Host Security Logs

In this post, I will be discussing an important and fundamental topic of information security. Namely, I will focus on the relative value differences between various sources of security data and why host security logs are at the top of my list when establishing a security program.

Anyone who has done some log diving and detection work knows that not all data are created equal. For security purposes, certain data types are far more effective for detection than others. This concept is important because security programs in many organizations must contend with constrained resources, thus increasing the importance of directing those resources in an optimal way.

This is true whether an organization has just begun bootstrapping their security program or whether they have extensive detection coverage and are just beginning to implement structured threat hunts.

Log Data Classes

Log data can be broken into classes or domains such as the following:

• Host

• Network

• Application

• Infrastructure/Middleware

• DNS/Proxy

Within each class of logs exist many different types that differ by platform.

Analyzing these log domains one at a time, it is interesting to consider what we can infer about their general detection usefulness based on the data we know they contain. Application logs can be helpful but generally only log specific actions within the application itself, can be very voluminous, and often only cover the initial compromise step of an intrusion. Additionally, this domain of logs generally only helps when web applications are the initial vector of compromise.

Infrastructure logs such as hypervisor, cloud, and middleware can be helpful for specific attack techniques, but generally sit too low on the stack for broad detection across all attack stages. The obvious exceptions to this are certain cloud-first/cloud-native applications which rely heavily on hosted Cloud services, serverless architecture, with properly configured access/object logging of those services. This assumes a lot though, since there are many organizations hosting applications in the cloud which took a "lift-and-shift" approach and whose configurations are far from perfect. For those organizations heavily utilizing traditional cloud virtual machines, cloud infrastructure/access logs fail to provide complete visibility.

Of course, DNS and proxy logs can be incredibly valuable in certain instances, but they suffer from narrow scope and prohibitive volume.

So Which Log Data Class is Best?

With all the limitations of the log classes mentioned above, I assert that the top two classes by value for general detection are host and network logs due to their broad scope and level of detail. Between these two, my highest priority is detailed host security events, which to me means a minimum of OS authentication events, detailed process events with command line parameters, and host network connection events. It's important to mention that these priorities could shift depending on many factors such as the needs of the organization, the architecture in question, and the types of threats an organization faces. It is very difficult to be dogmatic in a field with so many variables!

The Intuitive Case for Host Security Logs

Below are the intuitive reasons I generally prefer host logs over network logs:

• Lower Event Volume

• More Variation in Data Points makes it easier to distinguish legitimate/illegitimate behavior

• Proliferation of Network Encryption

Network volume, especially on medium to large sized network segments can be somewhat prohibitive from both the processing side as well as the alerting side. Life is a bit easier when a network is well documented and well segmented, but that increases the complexity of monitoring inter-host and inter-segment connections as well. While host network events probably generate an equivalent or greater volume of events as network flow events, host network events are more readily filtered and not all of them need to be shipped/centralized in real time. When it comes to more detailed network logs up to and including full Packet Capture, host network logs have exceptionally low volume in comparison.

To deal with the issue of volume when monitoring network traffic, one type of network logs, known as network flow logs, can be very useful. These logs are most often gathered on network sensors placed at network edges to collect inbound and outbound statistics on network traffic. Since they only gather metadata such as source/destination addresses and ports, they generate a reasonable volume of data while still offering a good deal of insight on network traffic. Advantages of this data type are relatively easier deployment of network sensors, no agents to deploy, no performance impact on hosts, and the ability to capture traffic obscured on a host/network device compromised with a rootkit (See Talos Blog on VPNFilter Malware).

The problem with these from a detection standpoint is lack of variation in data points. There are only so many combinations of IPs, ports, and bytes we can detect on. Statistically it can be effective to aggregate byte transfers or distinct addresses and ports by source, detect C2 beacons, and even detect anomalies within encrypted protocol tunnels (See SSH Inference Detection). However, smart adversaries can easily evade these by staying under the alert threshold and blending in with normal traffic, especially across network perimeters. Since adversaries generally know the traditional network perimeter is usually the best protected, odds are they can be much louder once they have established a foothold.

It could be argued that more sophisticated network security tools such as Tcpdump, Bro/Zeek, Suricata, Snort, and Splunk Stream grant far more depth of insight into the actual network traffic, and they can certainly be valuable when properly configured. The problem with this approach is the proliferation of encrypted protocols which block our ability to flag on byte/ascii strings, file hashes, etc. Even when encryption isn’t used, the ability to effectively and meaningfully analyze binaries transferred on the wire is prohibitive, leading us to often rely on backward-looking and easily defeated hash or string comparisons.

The Empirical Case for Host Security Logs

Now that I've covered the intuitive reasons, let's examine the empirical reasons host logs are generally superior from a cost versus benefit perspective for enterprise security monitoring:

• MITRE ATT&CK Detection Sources

• Security Product Reporting/Threat Intelligence

The most popular information security framework currently is the MITRE ATT&CK matrix which aims to document all known adversary techniques into atomic units and has gifted our community with a common method to communicate attack classes and techniques in a mutually intelligible way. Each adversary technique page in the matrix contains detailed descriptions, including data sources required for detection. Since this is a peer-reviewed framework, we can be confident in the data it contains.

When we use tools like Attackcti and Jupyter to explore the ATT&CK dataset, we can take a closer look and summarize the data sources which provide detection coverage for the most techniques. When we stack by data source, we see that detailed host process data detects the most documented techniques by far.

Number of MITRE ATT&CK Techniques Detectable by Log Data Class

In addition to community frameworks, threat intelligence reporting released to the public in 2019 and 2020 from high profile security product vendors such as Crowdstrike and Red Canary also guide us to host logging as the most valuable empirically. Although not all-encompassing, trends in security vendor reporting traverse many industry verticals and include both incidents discovered through their products as well as incidents investigated by their consulting services. Therefore, it’s reasonable to conclude they represent an excellent representative sample of attacks seen in the wild.

For me, the most convincing evidence from vendor threat reporting in favor of host security log monitoring comes from the 2019 Red Canary Threat Report. This vendor took the time to map each adversary behavior in 1774 Confirmed Threats to MITRE ATT&CK, stack-count the techniques, and order them based on prevalence across all the threats for the previous year. Below is the result.

MITRE ATT&CK Technique Prevalence in 2018. Source.

Although the exact numbers of incidents per technique weren't listed in the report, a cursory glance at the top 10 shows that process/command line logging would detect an overwhelming majority of the techniques per MITRE ATT&CK.

Additionally, the CrowdStrike Threat Report from 2019 confirms the widespread use of commodity malware and dual use/Living off the Land tools. Among the dual use and living off the land tools are PowerShell, Windows Management Instrumentation (WMI), Windows Scheduled Tasks, and PSExec. The Sophos 2020 Threat Report confirms several of these findings are still relevant today. Many of these tools already exist in the environment and are often used legitimately, so it is far more difficult to detect them based solely on their network traffic signatures. In this case, it is far better to focus on the precise behaviors being exhibited via process and command line monitoring.

Commodity malware such as PowerShell Empire, Cobalt Strike, and Covenant are used by more than just red teams for offensive operations. According to recent threat reports, adversaries have been relying on these tools heavily due to their widespread availability, powerful features and thorough documentation. Tools like Cobalt Strike's Malleable C2, when used properly, can be extremely difficult to detect on the wire via network data sources because it is designed to blend into normal network traffic patterns and generates very low traffic volume to defeat statistical detections.

The Downsides of Host Security Logs

Although host logs are an excellent data source for adversary detection, there are some downsides to consider when relying solely on these logs:

• Kernel Mode rootkits

• More difficult/slower to set up (agents, configuration, expensive, log centralization)

• Log Source Integrity (Renaming of EVTX files, tampering with ETW Trace Sessions which feed EVTX files, EDR agent vulnerabilities, etc.)

Summary

In summary, while there are many data sources available to network defenders to begin building their detection data pipelines, not all data sources have equivalent value. We must be analytical in our approach to decide where to begin exerting our energies, and carefully balance the unique needs of our organization with existing threat intelligence to help direct our resources to where they will deliver the biggest payoff. In the case of security logs for general enterprise defense, Host Security Logs, specifically Process and Command Line auditing, are an indispensable data source for real time detection. Although Network Flow and Packet Captures have tremendous value in real-time monitoring and are a great addition to security telemetry, the choice of which log data class to focus on first should be dictated by which class will yield the greatest detection coverage for the time spent.